2024-03-11T22:39:04.453 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-11T22:39:04.453 DEBUG:teuthology.run:Teuthology command: teuthology --description orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --name teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592012 --verbose --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592012/orig.config.yaml 2024-03-11T22:39:04.505 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-11T22:39:04.593 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592012 branch: squid description: orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7592012' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/quincy ' name: teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 94a539707733325322964c6a187c6a1b41be1daf ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 94a539707733325322964c6a187c6a1b41be1daf kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: fbcdfe674e56d1ea7a0810744e5915bb07961be5 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 3176 sha1: 94a539707733325322964c6a187c6a1b41be1daf sleep_before_teardown: 0 subset: 14/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: fbcdfe674e56d1ea7a0810744e5915bb07961be5 targets: smithi153.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGSU9hxjSkRd+UPEzk4H3VymcwJcjPDWVRnPgi+lAJGEQjMpSPys/9/UtPTw/AXe3s9bte0+1Ft3vbdE2ZqzApo= smithi163.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBr+LqZ81CgxZsYI9J50PTe8gyFZUpqQwnzD6uUcbAGvGTOHbEWwPUxcjZEYY+qazT5YfadfBUdAIuqqrVBb3Bw= tasks: - install: branch: quincy exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-11_21:08:02 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-03-11T22:39:04.593 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa; will attempt to use it 2024-03-11T22:39:04.594 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks 2024-03-11T22:39:04.594 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-11T22:39:04.596 INFO:teuthology.task.internal:Checking packages... 2024-03-11T22:39:04.616 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '94a539707733325322964c6a187c6a1b41be1daf' 2024-03-11T22:39:04.616 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-11T22:39:04.617 INFO:teuthology.packaging:ref: None 2024-03-11T22:39:04.617 INFO:teuthology.packaging:tag: None 2024-03-11T22:39:04.617 INFO:teuthology.packaging:branch: squid 2024-03-11T22:39:04.617 INFO:teuthology.packaging:sha1: 94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:39:04.617 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-11T22:39:04.884 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1607.g94a53970 2024-03-11T22:39:04.886 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-11T22:39:04.891 INFO:teuthology.task.internal:no buildpackages task found 2024-03-11T22:39:04.891 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-11T22:39:04.898 INFO:teuthology.task.internal:Saving configuration 2024-03-11T22:39:04.918 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-11T22:39:04.924 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T22:39:04.951 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592012', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 22:32:44.715334', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGSU9hxjSkRd+UPEzk4H3VymcwJcjPDWVRnPgi+lAJGEQjMpSPys/9/UtPTw/AXe3s9bte0+1Ft3vbdE2ZqzApo='} 2024-03-11T22:39:04.972 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi163.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592012', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 22:32:44.713856', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBr+LqZ81CgxZsYI9J50PTe8gyFZUpqQwnzD6uUcbAGvGTOHbEWwPUxcjZEYY+qazT5YfadfBUdAIuqqrVBb3Bw='} 2024-03-11T22:39:04.972 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-11T22:39:04.978 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-03-11T22:39:04.978 INFO:teuthology.task.internal:roles: ubuntu@smithi163.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-03-11T22:39:04.978 INFO:teuthology.run_tasks:Running task console_log... 2024-03-11T22:39:05.044 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9d402d60d0>, signals=[15]) 2024-03-11T22:39:05.045 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-11T22:39:05.051 INFO:teuthology.task.internal:Opening connections... 2024-03-11T22:39:05.051 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi153.front.sepia.ceph.com 2024-03-11T22:39:05.053 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:39:05.144 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi163.front.sepia.ceph.com 2024-03-11T22:39:05.144 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:39:05.230 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-11T22:39:05.236 DEBUG:teuthology.orchestra.run.smithi153:> uname -m 2024-03-11T22:39:05.253 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2024-03-11T22:39:05.253 DEBUG:teuthology.orchestra.run.smithi153:> cat /etc/os-release 2024-03-11T22:39:05.309 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="CentOS Stream" 2024-03-11T22:39:05.309 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="9" 2024-03-11T22:39:05.309 INFO:teuthology.orchestra.run.smithi153.stdout:ID="centos" 2024-03-11T22:39:05.309 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE="rhel fedora" 2024-03-11T22:39:05.309 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="9" 2024-03-11T22:39:05.309 INFO:teuthology.orchestra.run.smithi153.stdout:PLATFORM_ID="platform:el9" 2024-03-11T22:39:05.309 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T22:39:05.309 INFO:teuthology.orchestra.run.smithi153.stdout:ANSI_COLOR="0;31" 2024-03-11T22:39:05.309 INFO:teuthology.orchestra.run.smithi153.stdout:LOGO="fedora-logo-icon" 2024-03-11T22:39:05.309 INFO:teuthology.orchestra.run.smithi153.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T22:39:05.309 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://centos.org/" 2024-03-11T22:39:05.309 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T22:39:05.309 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T22:39:05.309 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T22:39:05.310 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2024-03-11T22:39:05.331 DEBUG:teuthology.orchestra.run.smithi163:> uname -m 2024-03-11T22:39:05.349 INFO:teuthology.orchestra.run.smithi163.stdout:x86_64 2024-03-11T22:39:05.349 DEBUG:teuthology.orchestra.run.smithi163:> cat /etc/os-release 2024-03-11T22:39:05.406 INFO:teuthology.orchestra.run.smithi163.stdout:NAME="CentOS Stream" 2024-03-11T22:39:05.406 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION="9" 2024-03-11T22:39:05.406 INFO:teuthology.orchestra.run.smithi163.stdout:ID="centos" 2024-03-11T22:39:05.406 INFO:teuthology.orchestra.run.smithi163.stdout:ID_LIKE="rhel fedora" 2024-03-11T22:39:05.406 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION_ID="9" 2024-03-11T22:39:05.406 INFO:teuthology.orchestra.run.smithi163.stdout:PLATFORM_ID="platform:el9" 2024-03-11T22:39:05.406 INFO:teuthology.orchestra.run.smithi163.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T22:39:05.406 INFO:teuthology.orchestra.run.smithi163.stdout:ANSI_COLOR="0;31" 2024-03-11T22:39:05.407 INFO:teuthology.orchestra.run.smithi163.stdout:LOGO="fedora-logo-icon" 2024-03-11T22:39:05.407 INFO:teuthology.orchestra.run.smithi163.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T22:39:05.407 INFO:teuthology.orchestra.run.smithi163.stdout:HOME_URL="https://centos.org/" 2024-03-11T22:39:05.407 INFO:teuthology.orchestra.run.smithi163.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T22:39:05.407 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T22:39:05.407 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T22:39:05.407 INFO:teuthology.lock.ops:Updating smithi163.front.sepia.ceph.com on lock server 2024-03-11T22:39:05.431 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-11T22:39:05.440 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-11T22:39:05.445 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-11T22:39:05.446 DEBUG:teuthology.orchestra.run.smithi153:> test '!' -e /home/ubuntu/cephtest 2024-03-11T22:39:05.448 DEBUG:teuthology.orchestra.run.smithi163:> test '!' -e /home/ubuntu/cephtest 2024-03-11T22:39:05.463 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-11T22:39:05.469 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-11T22:39:05.469 DEBUG:teuthology.orchestra.run.smithi153:> test -z $(ls -A /var/lib/ceph) 2024-03-11T22:39:05.504 DEBUG:teuthology.orchestra.run.smithi163:> test -z $(ls -A /var/lib/ceph) 2024-03-11T22:39:05.536 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-11T22:39:05.592 INFO:teuthology.run_tasks:Running task kernel... 2024-03-11T22:39:05.606 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-11T22:39:05.606 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-11T22:39:05.606 DEBUG:teuthology.orchestra.run.smithi153:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T22:39:05.606 DEBUG:teuthology.orchestra.run.smithi163:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T22:39:05.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:39:05.622 DEBUG:teuthology.orchestra.run.smithi163:> uname -r 2024-03-11T22:39:05.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:39:05.623 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-03-11T22:39:05.678 INFO:teuthology.orchestra.run.smithi163.stdout:5.14.0-361.el9.x86_64 2024-03-11T22:39:05.678 INFO:teuthology.task.kernel:Running kernel on smithi163: 5.14.0-361.el9.x86_64 2024-03-11T22:39:05.678 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum install -y kernel 2024-03-11T22:39:05.680 INFO:teuthology.orchestra.run.smithi153.stdout:5.14.0-361.el9.x86_64 2024-03-11T22:39:05.681 INFO:teuthology.task.kernel:Running kernel on smithi153: 5.14.0-361.el9.x86_64 2024-03-11T22:39:05.681 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum install -y kernel 2024-03-11T22:39:07.795 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-11T22:39:07.869 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-11T22:39:09.420 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-11T22:39:09.511 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-11T22:39:12.610 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - BaseOS 2.9 MB/s | 8.0 MB 00:02 2024-03-11T22:39:15.227 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-03-11T22:39:19.261 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-11T22:39:20.676 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 753 kB/s | 8.0 MB 00:10 2024-03-11T22:39:20.795 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 15 kB 00:00 2024-03-11T22:39:22.688 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-11T22:39:26.844 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 9.7 MB/s | 6.1 MB 00:00 2024-03-11T22:39:28.461 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 15 kB 00:00 2024-03-11T22:39:30.021 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-11T22:39:34.996 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-11T22:39:37.219 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T22:39:37.219 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T22:39:37.302 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-11T22:39:37.314 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-11T22:39:37.314 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-03-11T22:39:37.314 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-11T22:39:37.314 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-11T22:39:37.314 INFO:teuthology.orchestra.run.smithi153.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T22:39:37.314 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-11T22:39:37.314 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T22:39:37.315 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T22:39:37.315 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T22:39:37.315 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:39:37.315 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-11T22:39:37.315 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-11T22:39:37.315 INFO:teuthology.orchestra.run.smithi153.stdout:Install 4 Packages 2024-03-11T22:39:37.315 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:39:37.315 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 100 M 2024-03-11T22:39:37.315 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 123 M 2024-03-11T22:39:37.315 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-11T22:39:37.822 INFO:teuthology.orchestra.run.smithi153.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 14 MB/s | 6.0 MB 00:00 2024-03-11T22:39:38.523 INFO:teuthology.orchestra.run.smithi153.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 18 MB/s | 21 MB 00:01 2024-03-11T22:39:39.472 INFO:teuthology.orchestra.run.smithi153.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 20 MB/s | 33 MB 00:01 2024-03-11T22:39:39.823 INFO:teuthology.orchestra.run.smithi153.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 16 MB/s | 39 MB 00:02 2024-03-11T22:39:39.826 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:39:39.826 INFO:teuthology.orchestra.run.smithi153.stdout:Total 40 MB/s | 100 MB 00:02 2024-03-11T22:39:40.712 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-11T22:39:41.092 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-11T22:39:41.092 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-11T22:39:42.453 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-11T22:39:42.454 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-11T22:39:45.427 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-11T22:39:46.074 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T22:39:46.214 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:39:49.042 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:39:49.225 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:39:53.978 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:39:54.093 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:39:58.860 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:40:19.822 INFO:teuthology.orchestra.run.smithi163.stdout:Extra Packages for Enterprise Linux 358 kB/s | 21 MB 00:58 2024-03-11T22:40:24.605 INFO:teuthology.orchestra.run.smithi163.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-11T22:40:26.803 INFO:teuthology.orchestra.run.smithi163.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T22:40:26.803 INFO:teuthology.orchestra.run.smithi163.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T22:40:26.898 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-11T22:40:26.908 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-11T22:40:26.908 INFO:teuthology.orchestra.run.smithi163.stdout: Package Architecture Version Repository Size 2024-03-11T22:40:26.908 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-11T22:40:26.908 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-11T22:40:26.908 INFO:teuthology.orchestra.run.smithi163.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T22:40:26.908 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-03-11T22:40:26.908 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T22:40:26.908 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T22:40:26.908 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T22:40:26.908 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:40:26.908 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-11T22:40:26.909 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-11T22:40:26.909 INFO:teuthology.orchestra.run.smithi163.stdout:Install 4 Packages 2024-03-11T22:40:26.909 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:40:26.909 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 100 M 2024-03-11T22:40:26.909 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 123 M 2024-03-11T22:40:26.909 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-11T22:40:27.991 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:40:27.991 INFO:teuthology.orchestra.run.smithi153.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T22:40:27.992 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:40:28.078 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:40:28.505 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:40:28.506 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T22:40:28.506 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:40:28.506 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:40:28.933 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:40:28.933 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:40:28.933 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-11T22:40:28.934 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T22:40:28.934 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T22:40:28.934 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T22:40:28.934 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T22:40:28.934 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:40:28.934 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-11T22:40:29.411 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T22:40:29.559 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T22:40:29.559 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T22:40:29.559 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T22:40:29.559 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T22:40:29.560 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T22:40:29.560 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-03-11T22:40:29.560 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi153.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T22:40:29.560 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T22:40:29.681 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T22:40:29.681 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T22:40:29.682 DEBUG:teuthology.orchestra.run.smithi153:> sudo rpm -qi grub2-tools 2024-03-11T22:40:29.715 INFO:teuthology.orchestra.run.smithi153.stdout:Name : grub2-tools 2024-03-11T22:40:29.715 INFO:teuthology.orchestra.run.smithi153.stdout:Epoch : 1 2024-03-11T22:40:29.716 INFO:teuthology.orchestra.run.smithi153.stdout:Version : 2.06 2024-03-11T22:40:29.716 INFO:teuthology.orchestra.run.smithi153.stdout:Release : 68.el9 2024-03-11T22:40:29.716 INFO:teuthology.orchestra.run.smithi153.stdout:Architecture: x86_64 2024-03-11T22:40:29.716 INFO:teuthology.orchestra.run.smithi153.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T22:40:29.716 INFO:teuthology.orchestra.run.smithi153.stdout:Group : Unspecified 2024-03-11T22:40:29.716 INFO:teuthology.orchestra.run.smithi153.stdout:Size : 8264254 2024-03-11T22:40:29.716 INFO:teuthology.orchestra.run.smithi153.stdout:License : GPLv3+ 2024-03-11T22:40:29.716 INFO:teuthology.orchestra.run.smithi153.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T22:40:29.716 INFO:teuthology.orchestra.run.smithi153.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T22:40:29.716 INFO:teuthology.orchestra.run.smithi153.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T22:40:29.716 INFO:teuthology.orchestra.run.smithi153.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T22:40:29.716 INFO:teuthology.orchestra.run.smithi153.stdout:Packager : builder@centos.org 2024-03-11T22:40:29.716 INFO:teuthology.orchestra.run.smithi153.stdout:Vendor : CentOS 2024-03-11T22:40:29.716 INFO:teuthology.orchestra.run.smithi153.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T22:40:29.717 INFO:teuthology.orchestra.run.smithi153.stdout:Summary : Support tools for GRUB. 2024-03-11T22:40:29.717 INFO:teuthology.orchestra.run.smithi153.stdout:Description : 2024-03-11T22:40:29.717 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:40:29.717 INFO:teuthology.orchestra.run.smithi153.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T22:40:29.717 INFO:teuthology.orchestra.run.smithi153.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T22:40:29.717 INFO:teuthology.orchestra.run.smithi153.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T22:40:29.717 INFO:teuthology.orchestra.run.smithi153.stdout:hardware devices. 2024-03-11T22:40:29.717 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:40:29.717 INFO:teuthology.orchestra.run.smithi153.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T22:40:29.718 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T22:40:29.718 INFO:teuthology.task.kernel:Updating grub on smithi153 to boot 5.14.0-427.el9.x86_64 2024-03-11T22:40:29.718 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T22:40:30.340 INFO:teuthology.orchestra.run.smithi153.stderr:Generating grub configuration file ... 2024-03-11T22:40:31.718 INFO:teuthology.orchestra.run.smithi153.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T22:40:31.752 INFO:teuthology.orchestra.run.smithi153.stderr:done 2024-03-11T22:40:31.755 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2024-03-11T22:40:31.770 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.FFn2UOxfmg 2024-03-11T22:40:31.770 DEBUG:teuthology.orchestra.run.smithi153:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.FFn2UOxfmg 2024-03-11T22:40:31.838 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.FFn2UOxfmg 2024-03-11T22:40:31.981 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.FFn2UOxfmg is 6KB 2024-03-11T22:40:32.001 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.FFn2UOxfmg 2024-03-11T22:40:32.014 DEBUG:teuthology.orchestra.run.smithi153:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T22:40:32.082 INFO:teuthology.orchestra.run.smithi153.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T22:40:32.082 INFO:teuthology.orchestra.run.smithi153.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T22:40:32.083 INFO:teuthology.orchestra.run.smithi153.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T22:40:32.083 INFO:teuthology.orchestra.run.smithi153.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T22:40:32.084 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T22:40:32.237 DEBUG:teuthology.orchestra.run.smithi153:> sudo shutdown -r now 2024-03-11T22:40:37.230 INFO:teuthology.orchestra.run.smithi163.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 607 kB/s | 6.0 MB 00:10 2024-03-11T22:41:02.240 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-03-11T22:41:02.240 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-11T22:41:02.241 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:41:10.459 INFO:teuthology.orchestra.run.smithi163.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 492 kB/s | 21 MB 00:43 2024-03-11T22:41:36.802 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-03-11T22:41:45.804 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-11T22:41:45.806 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:41:49.306 INFO:teuthology.orchestra.run.smithi163.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 490 kB/s | 39 MB 01:22 2024-03-11T22:41:50.721 INFO:teuthology.orchestra.run.smithi163.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 466 kB/s | 33 MB 01:13 2024-03-11T22:41:50.721 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:41:50.721 INFO:teuthology.orchestra.run.smithi163.stdout:Total 1.2 MB/s | 100 MB 01:23 2024-03-11T22:41:51.600 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-11T22:41:52.003 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-11T22:41:52.003 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-11T22:41:53.650 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-11T22:41:53.650 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-11T22:41:56.574 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-11T22:41:57.240 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T22:41:57.372 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:42:00.171 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:42:00.365 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:42:04.190 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-03-11T22:42:05.099 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:42:05.226 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:42:09.957 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:42:16.191 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-11T22:42:16.192 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:42:16.556 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-03-11T22:42:17.074 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-11T22:42:17.074 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-427.el9.x86_64"... 2024-03-11T22:42:17.074 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-03-11T22:42:17.088 INFO:teuthology.orchestra.run.smithi153.stdout:5.14.0-427.el9.x86_64 2024-03-11T22:42:17.089 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T22:42:17.089 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T22:42:17.089 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T22:42:18.090 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-03-11T22:42:18.090 DEBUG:teuthology.orchestra.run.smithi153:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T22:42:18.176 INFO:teuthology.orchestra.run.smithi153.stdout:ttyS1 2024-03-11T22:42:18.222 DEBUG:teuthology.parallel:result is None 2024-03-11T22:42:39.147 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:42:39.147 INFO:teuthology.orchestra.run.smithi163.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T22:42:39.147 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:42:39.233 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:42:39.749 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:42:39.749 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T22:42:39.749 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:42:39.749 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:42:40.164 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:42:40.164 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:42:40.164 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-11T22:42:40.164 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T22:42:40.164 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T22:42:40.164 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T22:42:40.165 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T22:42:40.165 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:42:40.165 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-11T22:42:40.723 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T22:42:40.869 INFO:teuthology.orchestra.run.smithi163.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T22:42:40.870 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T22:42:40.870 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T22:42:40.870 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T22:42:40.870 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T22:42:40.870 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-03-11T22:42:40.870 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi163.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T22:42:40.870 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T22:42:40.992 INFO:teuthology.orchestra.run.smithi163.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T22:42:40.993 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T22:42:40.993 DEBUG:teuthology.orchestra.run.smithi163:> sudo rpm -qi grub2-tools 2024-03-11T22:42:41.030 INFO:teuthology.orchestra.run.smithi163.stdout:Name : grub2-tools 2024-03-11T22:42:41.030 INFO:teuthology.orchestra.run.smithi163.stdout:Epoch : 1 2024-03-11T22:42:41.031 INFO:teuthology.orchestra.run.smithi163.stdout:Version : 2.06 2024-03-11T22:42:41.031 INFO:teuthology.orchestra.run.smithi163.stdout:Release : 68.el9 2024-03-11T22:42:41.031 INFO:teuthology.orchestra.run.smithi163.stdout:Architecture: x86_64 2024-03-11T22:42:41.031 INFO:teuthology.orchestra.run.smithi163.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T22:42:41.031 INFO:teuthology.orchestra.run.smithi163.stdout:Group : Unspecified 2024-03-11T22:42:41.031 INFO:teuthology.orchestra.run.smithi163.stdout:Size : 8264254 2024-03-11T22:42:41.031 INFO:teuthology.orchestra.run.smithi163.stdout:License : GPLv3+ 2024-03-11T22:42:41.031 INFO:teuthology.orchestra.run.smithi163.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T22:42:41.031 INFO:teuthology.orchestra.run.smithi163.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T22:42:41.031 INFO:teuthology.orchestra.run.smithi163.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T22:42:41.031 INFO:teuthology.orchestra.run.smithi163.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T22:42:41.031 INFO:teuthology.orchestra.run.smithi163.stdout:Packager : builder@centos.org 2024-03-11T22:42:41.031 INFO:teuthology.orchestra.run.smithi163.stdout:Vendor : CentOS 2024-03-11T22:42:41.031 INFO:teuthology.orchestra.run.smithi163.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T22:42:41.031 INFO:teuthology.orchestra.run.smithi163.stdout:Summary : Support tools for GRUB. 2024-03-11T22:42:41.032 INFO:teuthology.orchestra.run.smithi163.stdout:Description : 2024-03-11T22:42:41.032 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:42:41.032 INFO:teuthology.orchestra.run.smithi163.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T22:42:41.032 INFO:teuthology.orchestra.run.smithi163.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T22:42:41.032 INFO:teuthology.orchestra.run.smithi163.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T22:42:41.032 INFO:teuthology.orchestra.run.smithi163.stdout:hardware devices. 2024-03-11T22:42:41.032 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:42:41.032 INFO:teuthology.orchestra.run.smithi163.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T22:42:41.033 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T22:42:41.033 INFO:teuthology.task.kernel:Updating grub on smithi163 to boot 5.14.0-427.el9.x86_64 2024-03-11T22:42:41.033 DEBUG:teuthology.orchestra.run.smithi163:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T22:42:41.621 INFO:teuthology.orchestra.run.smithi163.stderr:Generating grub configuration file ... 2024-03-11T22:42:43.165 INFO:teuthology.orchestra.run.smithi163.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T22:42:43.189 INFO:teuthology.orchestra.run.smithi163.stderr:done 2024-03-11T22:42:43.192 DEBUG:teuthology.orchestra.run.smithi163:> mktemp 2024-03-11T22:42:43.246 INFO:teuthology.orchestra.run.smithi163.stdout:/tmp/tmp.5F5NPQK9I4 2024-03-11T22:42:43.246 DEBUG:teuthology.orchestra.run.smithi163:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.5F5NPQK9I4 2024-03-11T22:42:43.315 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 0666 /tmp/tmp.5F5NPQK9I4 2024-03-11T22:42:43.455 DEBUG:teuthology.orchestra.remote:smithi163:/tmp/tmp.5F5NPQK9I4 is 6KB 2024-03-11T22:42:43.475 DEBUG:teuthology.orchestra.run.smithi163:> rm -fr /tmp/tmp.5F5NPQK9I4 2024-03-11T22:42:43.489 DEBUG:teuthology.orchestra.run.smithi163:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T22:42:43.553 INFO:teuthology.orchestra.run.smithi163.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T22:42:43.554 INFO:teuthology.orchestra.run.smithi163.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T22:42:43.554 INFO:teuthology.orchestra.run.smithi163.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T22:42:43.554 INFO:teuthology.orchestra.run.smithi163.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T22:42:43.555 DEBUG:teuthology.orchestra.run.smithi163:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T22:42:43.703 DEBUG:teuthology.orchestra.run.smithi163:> sudo shutdown -r now 2024-03-11T22:43:13.706 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-03-11T22:43:13.707 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T22:43:13.708 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:43:32.258 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T22:43:41.259 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T22:43:41.260 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:43:44.318 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T22:43:56.319 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T22:43:56.320 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:43:59.390 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T22:44:14.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T22:44:14.392 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:44:32.926 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T22:44:50.927 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T22:44:50.928 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:44:51.282 DEBUG:teuthology.orchestra.run.smithi163:> true 2024-03-11T22:44:51.821 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T22:44:51.821 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-427.el9.x86_64"... 2024-03-11T22:44:51.821 DEBUG:teuthology.orchestra.run.smithi163:> uname -r 2024-03-11T22:44:51.847 INFO:teuthology.orchestra.run.smithi163.stdout:5.14.0-427.el9.x86_64 2024-03-11T22:44:51.848 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T22:44:51.848 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T22:44:51.848 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T22:44:52.849 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-03-11T22:44:52.849 DEBUG:teuthology.orchestra.run.smithi163:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T22:44:52.919 INFO:teuthology.orchestra.run.smithi163.stdout:ttyS1 2024-03-11T22:44:52.952 DEBUG:teuthology.parallel:result is None 2024-03-11T22:44:52.952 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-11T22:44:52.964 INFO:teuthology.task.internal:Creating test directory... 2024-03-11T22:44:52.964 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T22:44:52.969 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T22:44:52.985 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-11T22:44:52.992 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-11T22:44:53.002 INFO:teuthology.task.internal:Creating archive directory... 2024-03-11T22:44:53.002 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T22:44:53.029 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T22:44:53.054 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-11T22:44:53.062 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-11T22:44:53.062 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T22:44:53.090 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T22:44:53.120 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T22:44:53.125 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T22:44:53.132 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T22:44:53.139 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T22:44:53.141 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-11T22:44:53.148 INFO:teuthology.task.internal:Configuring sudo... 2024-03-11T22:44:53.148 DEBUG:teuthology.orchestra.run.smithi153:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T22:44:53.183 DEBUG:teuthology.orchestra.run.smithi163:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T22:44:53.211 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-11T22:44:53.221 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-11T22:44:53.221 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T22:44:53.254 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T22:44:53.268 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T22:44:53.353 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T22:44:53.405 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T22:44:53.489 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T22:44:53.531 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-11T22:44:53.531 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T22:44:53.599 DEBUG:teuthology.orchestra.run.smithi163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T22:44:53.642 DEBUG:teuthology.orchestra.run.smithi163:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T22:44:53.691 DEBUG:teuthology.orchestra.run.smithi163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T22:44:53.774 DEBUG:teuthology.orchestra.run.smithi163:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T22:44:53.816 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-11T22:44:53.817 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T22:44:53.884 DEBUG:teuthology.orchestra.run.smithi153:> sudo service rsyslog restart 2024-03-11T22:44:53.887 DEBUG:teuthology.orchestra.run.smithi163:> sudo service rsyslog restart 2024-03-11T22:44:53.931 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T22:44:53.965 INFO:teuthology.orchestra.run.smithi163.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T22:44:54.487 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-11T22:44:54.496 INFO:teuthology.task.internal:Starting timer... 2024-03-11T22:44:54.496 INFO:teuthology.run_tasks:Running task pcp... 2024-03-11T22:44:54.571 INFO:teuthology.run_tasks:Running task selinux... 2024-03-11T22:44:54.582 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-11T22:44:54.582 DEBUG:teuthology.orchestra.run.smithi153:> sudo service auditd rotate 2024-03-11T22:44:54.644 INFO:teuthology.orchestra.run.smithi153.stdout:Rotating logs: 2024-03-11T22:44:54.645 DEBUG:teuthology.orchestra.run.smithi163:> sudo service auditd rotate 2024-03-11T22:44:54.703 INFO:teuthology.orchestra.run.smithi163.stdout:Rotating logs: 2024-03-11T22:44:54.705 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-11T22:44:54.705 DEBUG:teuthology.orchestra.run.smithi153:> /usr/sbin/getenforce 2024-03-11T22:44:54.726 INFO:teuthology.orchestra.run.smithi153.stdout:Permissive 2024-03-11T22:44:54.726 DEBUG:teuthology.orchestra.run.smithi163:> /usr/sbin/getenforce 2024-03-11T22:44:54.754 INFO:teuthology.orchestra.run.smithi163.stdout:Permissive 2024-03-11T22:44:54.754 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi153.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi163.front.sepia.ceph.com': 'permissive'} 2024-03-11T22:44:54.755 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T22:44:54.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:44:54.796 DEBUG:teuthology.orchestra.run.smithi163:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T22:44:54.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:44:54.824 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-11T22:44:54.824 DEBUG:teuthology.orchestra.run.smithi153:> sudo /usr/sbin/setenforce permissive 2024-03-11T22:44:54.870 DEBUG:teuthology.orchestra.run.smithi163:> sudo /usr/sbin/setenforce permissive 2024-03-11T22:44:54.904 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-11T22:44:54.919 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-11T22:44:54.926 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-11T22:44:55.071 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-11T22:44:55.094 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-11T22:44:55.096 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi153.front.sepia.ceph.com,smithi163.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-11T22:49:06.969 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi153.front.sepia.ceph.com'), Remote(name='ubuntu@smithi163.front.sepia.ceph.com')] 2024-03-11T22:49:06.971 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-11T22:49:06.972 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:49:07.064 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-03-11T22:49:07.141 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-11T22:49:07.142 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T22:49:07.142 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:49:07.228 DEBUG:teuthology.orchestra.run.smithi163:> true 2024-03-11T22:49:07.305 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T22:49:07.305 INFO:teuthology.run_tasks:Running task clock... 2024-03-11T22:49:07.317 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-11T22:49:07.317 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T22:49:07.317 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T22:49:07.320 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T22:49:07.320 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T22:49:07.348 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T22:49:07.366 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T22:49:07.377 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T22:49:07.393 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T22:49:07.395 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: ntpd: command not found 2024-03-11T22:49:07.409 INFO:teuthology.orchestra.run.smithi153.stdout:506 Cannot talk to daemon 2024-03-11T22:49:07.423 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: ntpd: command not found 2024-03-11T22:49:07.425 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T22:49:07.436 INFO:teuthology.orchestra.run.smithi163.stdout:506 Cannot talk to daemon 2024-03-11T22:49:07.442 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T22:49:07.453 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T22:49:07.469 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T22:49:07.493 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-03-11T22:49:07.496 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T22:49:07.497 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-03-11T22:49:07.497 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:49:07.497 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:49:07.497 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:49:07.497 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:49:07.523 INFO:teuthology.orchestra.run.smithi163.stderr:bash: line 1: ntpq: command not found 2024-03-11T22:49:07.526 INFO:teuthology.orchestra.run.smithi163.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T22:49:07.527 INFO:teuthology.orchestra.run.smithi163.stdout:=============================================================================== 2024-03-11T22:49:07.527 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:49:07.527 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:49:07.527 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:49:07.527 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:49:07.527 INFO:teuthology.run_tasks:Running task install... 2024-03-11T22:49:07.536 DEBUG:teuthology.task.install:project ceph 2024-03-11T22:49:07.536 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf'}} 2024-03-11T22:49:07.536 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf'} 2024-03-11T22:49:07.536 INFO:teuthology.task.install:Using flavor: default 2024-03-11T22:49:07.542 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-11T22:49:07.542 INFO:teuthology.task.install:extra packages: [] 2024-03-11T22:49:07.543 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'tag': None, 'wait_for_package': False} 2024-03-11T22:49:07.543 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-11T22:49:07.543 INFO:teuthology.packaging:ref: None 2024-03-11T22:49:07.543 INFO:teuthology.packaging:tag: None 2024-03-11T22:49:07.543 INFO:teuthology.packaging:branch: quincy 2024-03-11T22:49:07.543 INFO:teuthology.packaging:sha1: 94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:49:07.543 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-03-11T22:49:07.545 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'tag': None, 'wait_for_package': False} 2024-03-11T22:49:07.546 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-11T22:49:07.546 INFO:teuthology.packaging:ref: None 2024-03-11T22:49:07.546 INFO:teuthology.packaging:tag: None 2024-03-11T22:49:07.546 INFO:teuthology.packaging:branch: quincy 2024-03-11T22:49:07.546 INFO:teuthology.packaging:sha1: 94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:49:07.546 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-03-11T22:49:07.762 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/quincy/c186d0ca78be18052defec2e0c064f7869af6d3f/centos/9/flavors/default/ 2024-03-11T22:49:07.762 INFO:teuthology.task.install.rpm:Package version is 17.2.7-620.gc186d0ca 2024-03-11T22:49:07.781 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/quincy/c186d0ca78be18052defec2e0c064f7869af6d3f/centos/9/flavors/default/ 2024-03-11T22:49:07.782 INFO:teuthology.task.install.rpm:Package version is 17.2.7-620.gc186d0ca 2024-03-11T22:49:07.876 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/quincy/c186d0ca78be18052defec2e0c064f7869af6d3f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/c186d0ca78be18052defec2e0c064f7869af6d3f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/c186d0ca78be18052defec2e0c064f7869af6d3f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-11T22:49:07.876 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-11T22:49:07.876 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-11T22:49:07.905 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-11T22:49:07.905 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-11T22:49:07.905 INFO:teuthology.packaging:ref: None 2024-03-11T22:49:07.905 INFO:teuthology.packaging:tag: None 2024-03-11T22:49:07.905 INFO:teuthology.packaging:branch: quincy 2024-03-11T22:49:07.905 INFO:teuthology.packaging:sha1: 94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:49:07.905 DEBUG:teuthology.orchestra.run.smithi153:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-11T22:49:07.951 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/quincy/c186d0ca78be18052defec2e0c064f7869af6d3f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/c186d0ca78be18052defec2e0c064f7869af6d3f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/c186d0ca78be18052defec2e0c064f7869af6d3f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-11T22:49:07.951 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-11T22:49:07.951 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-11T22:49:07.972 DEBUG:teuthology.orchestra.run.smithi153:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-11T22:49:07.979 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-11T22:49:07.980 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-11T22:49:07.980 INFO:teuthology.packaging:ref: None 2024-03-11T22:49:07.980 INFO:teuthology.packaging:tag: None 2024-03-11T22:49:07.980 INFO:teuthology.packaging:branch: quincy 2024-03-11T22:49:07.980 INFO:teuthology.packaging:sha1: 94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:49:07.980 DEBUG:teuthology.orchestra.run.smithi163:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-11T22:49:08.047 DEBUG:teuthology.orchestra.run.smithi163:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-11T22:49:08.054 DEBUG:teuthology.orchestra.run.smithi153:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-11T22:49:08.120 INFO:teuthology.orchestra.run.smithi153.stdout:check_obsoletes = 1 2024-03-11T22:49:08.121 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum clean all 2024-03-11T22:49:08.127 DEBUG:teuthology.orchestra.run.smithi163:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-11T22:49:08.194 INFO:teuthology.orchestra.run.smithi163.stdout:check_obsoletes = 1 2024-03-11T22:49:08.195 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum clean all 2024-03-11T22:49:08.358 INFO:teuthology.orchestra.run.smithi153.stdout:74 files removed 2024-03-11T22:49:08.386 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-radosgw 2024-03-11T22:49:08.428 INFO:teuthology.orchestra.run.smithi163.stdout:74 files removed 2024-03-11T22:49:08.455 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-radosgw 2024-03-11T22:49:09.242 INFO:teuthology.orchestra.run.smithi153.stdout:ceph packages for x86_64 145 kB/s | 77 kB 00:00 2024-03-11T22:49:09.281 INFO:teuthology.orchestra.run.smithi163.stdout:ceph packages for x86_64 154 kB/s | 77 kB 00:00 2024-03-11T22:49:09.668 INFO:teuthology.orchestra.run.smithi153.stdout:ceph noarch packages 24 kB/s | 9.1 kB 00:00 2024-03-11T22:49:09.715 INFO:teuthology.orchestra.run.smithi163.stdout:ceph noarch packages 24 kB/s | 9.1 kB 00:00 2024-03-11T22:49:09.976 INFO:teuthology.orchestra.run.smithi153.stdout:ceph source packages 6.6 kB/s | 1.8 kB 00:00 2024-03-11T22:49:10.073 INFO:teuthology.orchestra.run.smithi163.stdout:ceph source packages 5.6 kB/s | 1.8 kB 00:00 2024-03-11T22:49:10.412 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-11T22:49:10.768 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-11T22:49:11.954 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - BaseOS 4.8 MB/s | 2.0 MB 00:00 2024-03-11T22:49:12.193 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS-9 - BaseOS 6.5 MB/s | 2.0 MB 00:00 2024-03-11T22:49:13.142 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-03-11T22:49:15.753 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 2.3 MB/s | 8.0 MB 00:03 2024-03-11T22:49:38.906 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - AppStream 781 kB/s | 19 MB 00:24 2024-03-11T22:49:40.676 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 809 kB/s | 19 MB 00:23 2024-03-11T22:49:42.973 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-11T22:49:44.526 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.1 MB 00:00 2024-03-11T22:49:44.550 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 15 kB 00:00 2024-03-11T22:49:46.344 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 15 kB 00:00 2024-03-11T22:50:04.750 INFO:teuthology.orchestra.run.smithi163.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 21 MB 00:20 2024-03-11T22:50:05.834 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 1.1 MB/s | 21 MB 00:19 2024-03-11T22:50:09.680 INFO:teuthology.orchestra.run.smithi163.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-03-11T22:50:10.776 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-11T22:50:12.249 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-11T22:50:12.257 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-11T22:50:12.257 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-03-11T22:50:12.257 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-11T22:50:12.257 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-11T22:50:12.258 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-radosgw x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 14 M 2024-03-11T22:50:12.258 INFO:teuthology.orchestra.run.smithi163.stdout:Upgrading: 2024-03-11T22:50:12.258 INFO:teuthology.orchestra.run.smithi163.stdout: librados2 x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 3.1 M 2024-03-11T22:50:12.258 INFO:teuthology.orchestra.run.smithi163.stdout: librbd1 x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 2.9 M 2024-03-11T22:50:12.258 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-03-11T22:50:12.258 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-base x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 5.0 M 2024-03-11T22:50:12.258 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-common x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 23 M 2024-03-11T22:50:12.258 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-selinux x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 25 k 2024-03-11T22:50:12.258 INFO:teuthology.orchestra.run.smithi163.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-11T22:50:12.258 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs2 x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 635 k 2024-03-11T22:50:12.258 INFO:teuthology.orchestra.run.smithi163.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-11T22:50:12.258 INFO:teuthology.orchestra.run.smithi163.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-11T22:50:12.259 INFO:teuthology.orchestra.run.smithi163.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-11T22:50:12.259 INFO:teuthology.orchestra.run.smithi163.stdout: libradosstriper1 x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 457 k 2024-03-11T22:50:12.259 INFO:teuthology.orchestra.run.smithi163.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-11T22:50:12.259 INFO:teuthology.orchestra.run.smithi163.stdout: librgw2 x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 4.9 M 2024-03-11T22:50:12.259 INFO:teuthology.orchestra.run.smithi163.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-11T22:50:12.259 INFO:teuthology.orchestra.run.smithi163.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-11T22:50:12.259 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-argparse x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 45 k 2024-03-11T22:50:12.259 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-common x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 116 k 2024-03-11T22:50:12.259 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cephfs x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 156 k 2024-03-11T22:50:12.259 INFO:teuthology.orchestra.run.smithi163.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-11T22:50:12.259 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rados x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 320 k 2024-03-11T22:50:12.259 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rbd x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 292 k 2024-03-11T22:50:12.259 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rgw x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 99 k 2024-03-11T22:50:12.259 INFO:teuthology.orchestra.run.smithi163.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-11T22:50:12.259 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:50:12.260 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-11T22:50:12.260 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-11T22:50:12.260 INFO:teuthology.orchestra.run.smithi163.stdout:Install 22 Packages 2024-03-11T22:50:12.260 INFO:teuthology.orchestra.run.smithi163.stdout:Upgrade 2 Packages 2024-03-11T22:50:12.260 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:50:12.260 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 58 M 2024-03-11T22:50:12.260 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-11T22:50:13.327 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-11T22:50:13.340 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-11T22:50:13.340 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-11T22:50:13.341 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-11T22:50:13.341 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-11T22:50:13.341 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 14 M 2024-03-11T22:50:13.341 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrading: 2024-03-11T22:50:13.341 INFO:teuthology.orchestra.run.smithi153.stdout: librados2 x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 3.1 M 2024-03-11T22:50:13.341 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1 x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 2.9 M 2024-03-11T22:50:13.341 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-11T22:50:13.341 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 5.0 M 2024-03-11T22:50:13.341 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 23 M 2024-03-11T22:50:13.341 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 25 k 2024-03-11T22:50:13.341 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-11T22:50:13.341 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2 x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 635 k 2024-03-11T22:50:13.341 INFO:teuthology.orchestra.run.smithi153.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-11T22:50:13.341 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-11T22:50:13.342 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-11T22:50:13.342 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1 x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 457 k 2024-03-11T22:50:13.342 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-11T22:50:13.342 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2 x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 4.9 M 2024-03-11T22:50:13.342 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-11T22:50:13.342 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-11T22:50:13.342 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 45 k 2024-03-11T22:50:13.342 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 116 k 2024-03-11T22:50:13.342 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 156 k 2024-03-11T22:50:13.342 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-11T22:50:13.342 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 320 k 2024-03-11T22:50:13.342 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 292 k 2024-03-11T22:50:13.342 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 99 k 2024-03-11T22:50:13.342 INFO:teuthology.orchestra.run.smithi153.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-11T22:50:13.343 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:50:13.343 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-11T22:50:13.343 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-11T22:50:13.343 INFO:teuthology.orchestra.run.smithi153.stdout:Install 22 Packages 2024-03-11T22:50:13.343 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrade 2 Packages 2024-03-11T22:50:13.343 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:50:13.343 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 58 M 2024-03-11T22:50:13.343 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-11T22:50:14.048 INFO:teuthology.orchestra.run.smithi163.stdout:(1/24): ceph-base-17.2.7-620.gc186d0ca.el9.x86_ 2.9 MB/s | 5.0 MB 00:01 2024-03-11T22:50:14.148 INFO:teuthology.orchestra.run.smithi163.stdout:(2/24): ceph-selinux-17.2.7-620.gc186d0ca.el9.x 252 kB/s | 25 kB 00:00 2024-03-11T22:50:14.299 INFO:teuthology.orchestra.run.smithi163.stdout:(3/24): ceph-radosgw-17.2.7-620.gc186d0ca.el9.x 7.1 MB/s | 14 MB 00:01 2024-03-11T22:50:14.332 INFO:teuthology.orchestra.run.smithi163.stdout:(4/24): libcephfs2-17.2.7-620.gc186d0ca.el9.x86 3.4 MB/s | 635 kB 00:00 2024-03-11T22:50:14.399 INFO:teuthology.orchestra.run.smithi163.stdout:(5/24): libradosstriper1-17.2.7-620.gc186d0ca.e 4.5 MB/s | 457 kB 00:00 2024-03-11T22:50:14.483 INFO:teuthology.orchestra.run.smithi163.stdout:(6/24): python3-ceph-argparse-17.2.7-620.gc186d 542 kB/s | 45 kB 00:00 2024-03-11T22:50:14.566 INFO:teuthology.orchestra.run.smithi163.stdout:(7/24): python3-ceph-common-17.2.7-620.gc186d0c 1.4 MB/s | 116 kB 00:00 2024-03-11T22:50:14.775 INFO:teuthology.orchestra.run.smithi163.stdout:(8/24): ceph-common-17.2.7-620.gc186d0ca.el9.x8 9.2 MB/s | 23 MB 00:02 2024-03-11T22:50:14.800 INFO:teuthology.orchestra.run.smithi163.stdout:(9/24): python3-cephfs-17.2.7-620.gc186d0ca.el9 669 kB/s | 156 kB 00:00 2024-03-11T22:50:14.842 INFO:teuthology.orchestra.run.smithi163.stdout:(10/24): python3-rados-17.2.7-620.gc186d0ca.el9 4.7 MB/s | 320 kB 00:00 2024-03-11T22:50:14.867 INFO:teuthology.orchestra.run.smithi163.stdout:(11/24): python3-rbd-17.2.7-620.gc186d0ca.el9.x 4.3 MB/s | 292 kB 00:00 2024-03-11T22:50:14.968 INFO:teuthology.orchestra.run.smithi163.stdout:(12/24): python3-rgw-17.2.7-620.gc186d0ca.el9.x 793 kB/s | 99 kB 00:00 2024-03-11T22:50:15.110 INFO:teuthology.orchestra.run.smithi163.stdout:(13/24): librgw2-17.2.7-620.gc186d0ca.el9.x86_6 6.3 MB/s | 4.9 MB 00:00 2024-03-11T22:50:15.210 INFO:teuthology.orchestra.run.smithi163.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 465 kB/s | 159 kB 00:00 2024-03-11T22:50:15.235 INFO:teuthology.orchestra.run.smithi163.stdout:(15/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 184 kB/s | 44 kB 00:00 2024-03-11T22:50:15.335 INFO:teuthology.orchestra.run.smithi163.stdout:(16/24): librdkafka-1.6.1-102.el9.x86_64.rpm 3.3 MB/s | 661 kB 00:00 2024-03-11T22:50:15.360 INFO:teuthology.orchestra.run.smithi163.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.8 MB/s | 282 kB 00:00 2024-03-11T22:50:15.386 INFO:teuthology.orchestra.run.smithi163.stdout:(18/24): python3-prettytable-0.7.2-27.el9.noarc 273 kB/s | 41 kB 00:00 2024-03-11T22:50:15.444 INFO:teuthology.orchestra.run.smithi153.stdout:(1/24): ceph-base-17.2.7-620.gc186d0ca.el9.x86_ 2.5 MB/s | 5.0 MB 00:02 2024-03-11T22:50:15.502 INFO:teuthology.orchestra.run.smithi153.stdout:(2/24): ceph-selinux-17.2.7-620.gc186d0ca.el9.x 433 kB/s | 25 kB 00:00 2024-03-11T22:50:15.619 INFO:teuthology.orchestra.run.smithi153.stdout:(3/24): libcephfs2-17.2.7-620.gc186d0ca.el9.x86 5.3 MB/s | 635 kB 00:00 2024-03-11T22:50:15.669 INFO:teuthology.orchestra.run.smithi163.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 158 kB/s | 49 kB 00:00 2024-03-11T22:50:15.711 INFO:teuthology.orchestra.run.smithi163.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 207 kB/s | 67 kB 00:00 2024-03-11T22:50:15.719 INFO:teuthology.orchestra.run.smithi153.stdout:(4/24): libradosstriper1-17.2.7-620.gc186d0ca.e 4.5 MB/s | 457 kB 00:00 2024-03-11T22:50:15.778 INFO:teuthology.orchestra.run.smithi163.stdout:(21/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 698 kB/s | 309 kB 00:00 2024-03-11T22:50:15.995 INFO:teuthology.orchestra.run.smithi163.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 4.9 MB/s | 1.6 MB 00:00 2024-03-11T22:50:16.462 INFO:teuthology.orchestra.run.smithi163.stdout:(23/24): librbd1-17.2.7-620.gc186d0ca.el9.x86_6 4.3 MB/s | 2.9 MB 00:00 2024-03-11T22:50:16.521 INFO:teuthology.orchestra.run.smithi163.stdout:(24/24): librados2-17.2.7-620.gc186d0ca.el9.x86 3.8 MB/s | 3.1 MB 00:00 2024-03-11T22:50:16.529 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:50:16.529 INFO:teuthology.orchestra.run.smithi163.stdout:Total 14 MB/s | 58 MB 00:04 2024-03-11T22:50:16.529 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-11T22:50:16.563 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-11T22:50:16.563 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-11T22:50:16.603 INFO:teuthology.orchestra.run.smithi153.stdout:(5/24): librgw2-17.2.7-620.gc186d0ca.el9.x86_64 5.6 MB/s | 4.9 MB 00:00 2024-03-11T22:50:16.737 INFO:teuthology.orchestra.run.smithi153.stdout:(6/24): ceph-radosgw-17.2.7-620.gc186d0ca.el9.x 4.2 MB/s | 14 MB 00:03 2024-03-11T22:50:16.770 INFO:teuthology.orchestra.run.smithi153.stdout:(7/24): python3-ceph-argparse-17.2.7-620.gc186d 270 kB/s | 45 kB 00:00 2024-03-11T22:50:16.804 INFO:teuthology.orchestra.run.smithi153.stdout:(8/24): python3-ceph-common-17.2.7-620.gc186d0c 1.7 MB/s | 116 kB 00:00 2024-03-11T22:50:16.830 INFO:teuthology.orchestra.run.smithi153.stdout:(9/24): python3-cephfs-17.2.7-620.gc186d0ca.el9 2.6 MB/s | 156 kB 00:00 2024-03-11T22:50:16.864 INFO:teuthology.orchestra.run.smithi153.stdout:(10/24): python3-rados-17.2.7-620.gc186d0ca.el9 5.3 MB/s | 320 kB 00:00 2024-03-11T22:50:16.905 INFO:teuthology.orchestra.run.smithi153.stdout:(11/24): python3-rbd-17.2.7-620.gc186d0ca.el9.x 3.8 MB/s | 292 kB 00:00 2024-03-11T22:50:16.939 INFO:teuthology.orchestra.run.smithi153.stdout:(12/24): python3-rgw-17.2.7-620.gc186d0ca.el9.x 1.3 MB/s | 99 kB 00:00 2024-03-11T22:50:17.028 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-11T22:50:17.029 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-11T22:50:17.065 INFO:teuthology.orchestra.run.smithi153.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 353 kB/s | 44 kB 00:00 2024-03-11T22:50:17.106 INFO:teuthology.orchestra.run.smithi153.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 794 kB/s | 159 kB 00:00 2024-03-11T22:50:17.173 INFO:teuthology.orchestra.run.smithi153.stdout:(15/24): librdkafka-1.6.1-102.el9.x86_64.rpm 6.0 MB/s | 661 kB 00:00 2024-03-11T22:50:17.198 INFO:teuthology.orchestra.run.smithi153.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-03-11T22:50:17.216 INFO:teuthology.orchestra.run.smithi153.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 989 kB/s | 41 kB 00:00 2024-03-11T22:50:17.274 INFO:teuthology.orchestra.run.smithi153.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 834 kB/s | 49 kB 00:00 2024-03-11T22:50:17.315 INFO:teuthology.orchestra.run.smithi153.stdout:(19/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.6 MB/s | 309 kB 00:00 2024-03-11T22:50:17.340 INFO:teuthology.orchestra.run.smithi153.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-11T22:50:17.399 INFO:teuthology.orchestra.run.smithi153.stdout:(21/24): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-03-11T22:50:17.600 INFO:teuthology.orchestra.run.smithi153.stdout:(22/24): ceph-common-17.2.7-620.gc186d0ca.el9.x 5.4 MB/s | 23 MB 00:04 2024-03-11T22:50:17.808 INFO:teuthology.orchestra.run.smithi153.stdout:(23/24): librbd1-17.2.7-620.gc186d0ca.el9.x86_6 7.2 MB/s | 2.9 MB 00:00 2024-03-11T22:50:17.858 INFO:teuthology.orchestra.run.smithi153.stdout:(24/24): librados2-17.2.7-620.gc186d0ca.el9.x86 5.9 MB/s | 3.1 MB 00:00 2024-03-11T22:50:17.862 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:50:17.862 INFO:teuthology.orchestra.run.smithi153.stdout:Total 13 MB/s | 58 MB 00:04 2024-03-11T22:50:17.862 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-11T22:50:17.892 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-11T22:50:17.892 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-11T22:50:18.285 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-11T22:50:18.353 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-11T22:50:18.353 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-11T22:50:18.626 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-03-11T22:50:18.689 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : librados2-2:17.2.7-620.gc186d0ca.el9.x86_64 2/26 2024-03-11T22:50:18.766 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados2-2:17.2.7-620.gc186d0ca.el9.x86_64 2/26 2024-03-11T22:50:18.843 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rados-2:17.2.7-620.gc186d0ca.el9.x86_64 3/26 2024-03-11T22:50:18.882 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephfs2-2:17.2.7-620.gc186d0ca.el9.x86_64 4/26 2024-03-11T22:50:18.954 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephfs2-2:17.2.7-620.gc186d0ca.el9.x86_64 4/26 2024-03-11T22:50:19.008 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-03-11T22:50:19.045 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-03-11T22:50:19.481 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-03-11T22:50:19.510 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librgw2-2:17.2.7-620.gc186d0ca.el9.x86_64 8/26 2024-03-11T22:50:19.556 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librgw2-2:17.2.7-620.gc186d0ca.el9.x86_64 8/26 2024-03-11T22:50:19.584 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-03-11T22:50:19.627 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ceph-argparse-2:17.2.7-620.gc186d0ca.el9.x 10/26 2024-03-11T22:50:19.665 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cephfs-2:17.2.7-620.gc186d0ca.el9.x86_64 11/26 2024-03-11T22:50:19.727 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rgw-2:17.2.7-620.gc186d0ca.el9.x86_64 12/26 2024-03-11T22:50:19.754 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libradosstriper1-2:17.2.7-620.gc186d0ca.el9.x86_64 13/26 2024-03-11T22:50:19.767 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-11T22:50:19.846 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libradosstriper1-2:17.2.7-620.gc186d0ca.el9.x86_64 13/26 2024-03-11T22:50:19.885 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-03-11T22:50:19.921 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-03-11T22:50:19.953 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-03-11T22:50:20.140 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-03-11T22:50:20.177 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librados2-2:17.2.7-620.gc186d0ca.el9.x86_64 2/26 2024-03-11T22:50:20.263 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-03-11T22:50:20.271 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:17.2.7-620.gc186d0ca.el9.x86_64 2/26 2024-03-11T22:50:20.299 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : librbd1-2:17.2.7-620.gc186d0ca.el9.x86_64 18/26 2024-03-11T22:50:20.348 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rados-2:17.2.7-620.gc186d0ca.el9.x86_64 3/26 2024-03-11T22:50:20.371 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librbd1-2:17.2.7-620.gc186d0ca.el9.x86_64 18/26 2024-03-11T22:50:20.387 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs2-2:17.2.7-620.gc186d0ca.el9.x86_64 4/26 2024-03-11T22:50:20.437 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rbd-2:17.2.7-620.gc186d0ca.el9.x86_64 19/26 2024-03-11T22:50:20.460 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ceph-common-2:17.2.7-620.gc186d0ca.el9.x86 20/26 2024-03-11T22:50:20.476 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs2-2:17.2.7-620.gc186d0ca.el9.x86_64 4/26 2024-03-11T22:50:20.530 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-03-11T22:50:20.567 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-03-11T22:50:21.001 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-03-11T22:50:21.023 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librgw2-2:17.2.7-620.gc186d0ca.el9.x86_64 8/26 2024-03-11T22:50:21.069 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librgw2-2:17.2.7-620.gc186d0ca.el9.x86_64 8/26 2024-03-11T22:50:21.098 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-03-11T22:50:21.132 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-argparse-2:17.2.7-620.gc186d0ca.el9.x 10/26 2024-03-11T22:50:21.170 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cephfs-2:17.2.7-620.gc186d0ca.el9.x86_64 11/26 2024-03-11T22:50:21.232 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rgw-2:17.2.7-620.gc186d0ca.el9.x86_64 12/26 2024-03-11T22:50:21.259 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libradosstriper1-2:17.2.7-620.gc186d0ca.el9.x86_64 13/26 2024-03-11T22:50:21.309 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libradosstriper1-2:17.2.7-620.gc186d0ca.el9.x86_64 13/26 2024-03-11T22:50:21.349 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-03-11T22:50:21.384 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-03-11T22:50:21.417 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-03-11T22:50:21.713 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-03-11T22:50:21.739 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librbd1-2:17.2.7-620.gc186d0ca.el9.x86_64 18/26 2024-03-11T22:50:21.810 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:17.2.7-620.gc186d0ca.el9.x86_64 18/26 2024-03-11T22:50:21.885 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rbd-2:17.2.7-620.gc186d0ca.el9.x86_64 19/26 2024-03-11T22:50:21.916 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-common-2:17.2.7-620.gc186d0ca.el9.x86 20/26 2024-03-11T22:50:22.810 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-common-2:17.2.7-620.gc186d0ca.el9.x86_64 21/26 2024-03-11T22:50:22.856 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-common-2:17.2.7-620.gc186d0ca.el9.x86_64 21/26 2024-03-11T22:50:23.400 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-common-2:17.2.7-620.gc186d0ca.el9.x86_64 21/26 2024-03-11T22:50:23.434 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-base-2:17.2.7-620.gc186d0ca.el9.x86_64 22/26 2024-03-11T22:50:23.513 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-base-2:17.2.7-620.gc186d0ca.el9.x86_64 22/26 2024-03-11T22:50:23.513 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-11T22:50:23.513 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-11T22:50:23.514 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:50:23.552 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-selinux-2:17.2.7-620.gc186d0ca.el9.x86_64 23/26 2024-03-11T22:50:24.288 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:17.2.7-620.gc186d0ca.el9.x86_64 21/26 2024-03-11T22:50:24.321 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-common-2:17.2.7-620.gc186d0ca.el9.x86_64 21/26 2024-03-11T22:50:24.864 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:17.2.7-620.gc186d0ca.el9.x86_64 21/26 2024-03-11T22:50:24.899 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-base-2:17.2.7-620.gc186d0ca.el9.x86_64 22/26 2024-03-11T22:50:24.991 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-base-2:17.2.7-620.gc186d0ca.el9.x86_64 22/26 2024-03-11T22:50:24.991 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-11T22:50:24.992 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-11T22:50:24.992 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:50:25.026 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-selinux-2:17.2.7-620.gc186d0ca.el9.x86_64 23/26 2024-03-11T22:50:32.976 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-selinux-2:17.2.7-620.gc186d0ca.el9.x86_64 23/26 2024-03-11T22:50:32.977 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /sys 2024-03-11T22:50:32.977 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /proc 2024-03-11T22:50:32.977 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /mnt 2024-03-11T22:50:32.977 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /var/tmp 2024-03-11T22:50:32.977 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /home 2024-03-11T22:50:32.977 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /root 2024-03-11T22:50:32.977 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /tmp 2024-03-11T22:50:32.977 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:50:34.284 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-radosgw-2:17.2.7-620.gc186d0ca.el9.x86_64 24/26 2024-03-11T22:50:34.345 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-620.gc186d0ca.el9.x86_64 24/26 2024-03-11T22:50:34.345 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:50:34.345 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-11T22:50:34.345 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T22:50:34.345 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T22:50:34.345 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:50:34.346 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-03-11T22:50:34.403 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-selinux-2:17.2.7-620.gc186d0ca.el9.x86_64 23/26 2024-03-11T22:50:34.403 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /sys 2024-03-11T22:50:34.403 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /proc 2024-03-11T22:50:34.403 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /mnt 2024-03-11T22:50:34.403 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /var/tmp 2024-03-11T22:50:34.403 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /home 2024-03-11T22:50:34.403 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /root 2024-03-11T22:50:34.403 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /tmp 2024-03-11T22:50:34.404 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:50:34.437 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-03-11T22:50:34.437 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-03-11T22:50:35.700 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-radosgw-2:17.2.7-620.gc186d0ca.el9.x86_64 24/26 2024-03-11T22:50:35.713 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-03-11T22:50:35.713 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-base-2:17.2.7-620.gc186d0ca.el9.x86_64 1/26 2024-03-11T22:50:35.713 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-common-2:17.2.7-620.gc186d0ca.el9.x86_64 2/26 2024-03-11T22:50:35.713 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-radosgw-2:17.2.7-620.gc186d0ca.el9.x86_64 3/26 2024-03-11T22:50:35.713 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-selinux-2:17.2.7-620.gc186d0ca.el9.x86_64 4/26 2024-03-11T22:50:35.713 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephfs2-2:17.2.7-620.gc186d0ca.el9.x86_64 5/26 2024-03-11T22:50:35.713 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libradosstriper1-2:17.2.7-620.gc186d0ca.el9.x86_64 6/26 2024-03-11T22:50:35.713 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librgw2-2:17.2.7-620.gc186d0ca.el9.x86_64 7/26 2024-03-11T22:50:35.713 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ceph-argparse-2:17.2.7-620.gc186d0ca.el9.x 8/26 2024-03-11T22:50:35.713 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ceph-common-2:17.2.7-620.gc186d0ca.el9.x86 9/26 2024-03-11T22:50:35.714 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cephfs-2:17.2.7-620.gc186d0ca.el9.x86_64 10/26 2024-03-11T22:50:35.714 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rados-2:17.2.7-620.gc186d0ca.el9.x86_64 11/26 2024-03-11T22:50:35.714 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rbd-2:17.2.7-620.gc186d0ca.el9.x86_64 12/26 2024-03-11T22:50:35.714 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rgw-2:17.2.7-620.gc186d0ca.el9.x86_64 13/26 2024-03-11T22:50:35.714 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-03-11T22:50:35.714 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-03-11T22:50:35.714 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-03-11T22:50:35.714 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-03-11T22:50:35.714 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-03-11T22:50:35.714 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-03-11T22:50:35.714 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-03-11T22:50:35.714 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-03-11T22:50:35.714 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-03-11T22:50:35.714 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados2-2:17.2.7-620.gc186d0ca.el9.x86_64 23/26 2024-03-11T22:50:35.715 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-03-11T22:50:35.715 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librbd1-2:17.2.7-620.gc186d0ca.el9.x86_64 25/26 2024-03-11T22:50:35.761 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-620.gc186d0ca.el9.x86_64 24/26 2024-03-11T22:50:35.761 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:50:35.761 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-11T22:50:35.761 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T22:50:35.761 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T22:50:35.761 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:50:35.763 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-03-11T22:50:35.861 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-03-11T22:50:35.861 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-03-11T22:50:36.668 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-03-11T22:50:36.668 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:50:36.668 INFO:teuthology.orchestra.run.smithi163.stdout:Upgraded: 2024-03-11T22:50:36.668 INFO:teuthology.orchestra.run.smithi163.stdout: librados2-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:50:36.668 INFO:teuthology.orchestra.run.smithi163.stdout: librbd1-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:50:36.668 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-11T22:50:36.668 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-base-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:50:36.668 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-common-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:50:36.669 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-radosgw-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:50:36.669 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-selinux-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:50:36.669 INFO:teuthology.orchestra.run.smithi163.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-11T22:50:36.669 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs2-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:50:36.669 INFO:teuthology.orchestra.run.smithi163.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-11T22:50:36.669 INFO:teuthology.orchestra.run.smithi163.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-11T22:50:36.669 INFO:teuthology.orchestra.run.smithi163.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-11T22:50:36.669 INFO:teuthology.orchestra.run.smithi163.stdout: libradosstriper1-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:50:36.669 INFO:teuthology.orchestra.run.smithi163.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-11T22:50:36.669 INFO:teuthology.orchestra.run.smithi163.stdout: librgw2-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:50:36.669 INFO:teuthology.orchestra.run.smithi163.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-11T22:50:36.669 INFO:teuthology.orchestra.run.smithi163.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-11T22:50:36.669 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-argparse-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:50:36.669 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-common-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:50:36.670 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cephfs-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:50:36.670 INFO:teuthology.orchestra.run.smithi163.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-11T22:50:36.670 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rados-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:50:36.670 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rbd-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:50:36.670 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rgw-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:50:36.670 INFO:teuthology.orchestra.run.smithi163.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-11T22:50:36.670 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:50:36.670 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-11T22:50:37.057 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-test 2024-03-11T22:50:37.085 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-03-11T22:50:37.086 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-base-2:17.2.7-620.gc186d0ca.el9.x86_64 1/26 2024-03-11T22:50:37.086 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-common-2:17.2.7-620.gc186d0ca.el9.x86_64 2/26 2024-03-11T22:50:37.086 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-radosgw-2:17.2.7-620.gc186d0ca.el9.x86_64 3/26 2024-03-11T22:50:37.086 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-selinux-2:17.2.7-620.gc186d0ca.el9.x86_64 4/26 2024-03-11T22:50:37.086 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs2-2:17.2.7-620.gc186d0ca.el9.x86_64 5/26 2024-03-11T22:50:37.086 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libradosstriper1-2:17.2.7-620.gc186d0ca.el9.x86_64 6/26 2024-03-11T22:50:37.086 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librgw2-2:17.2.7-620.gc186d0ca.el9.x86_64 7/26 2024-03-11T22:50:37.086 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-argparse-2:17.2.7-620.gc186d0ca.el9.x 8/26 2024-03-11T22:50:37.086 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-common-2:17.2.7-620.gc186d0ca.el9.x86 9/26 2024-03-11T22:50:37.087 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cephfs-2:17.2.7-620.gc186d0ca.el9.x86_64 10/26 2024-03-11T22:50:37.087 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rados-2:17.2.7-620.gc186d0ca.el9.x86_64 11/26 2024-03-11T22:50:37.087 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rbd-2:17.2.7-620.gc186d0ca.el9.x86_64 12/26 2024-03-11T22:50:37.087 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rgw-2:17.2.7-620.gc186d0ca.el9.x86_64 13/26 2024-03-11T22:50:37.087 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-03-11T22:50:37.087 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-03-11T22:50:37.087 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-03-11T22:50:37.087 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-03-11T22:50:37.087 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-03-11T22:50:37.087 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-03-11T22:50:37.087 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-03-11T22:50:37.087 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-03-11T22:50:37.087 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-03-11T22:50:37.087 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:17.2.7-620.gc186d0ca.el9.x86_64 23/26 2024-03-11T22:50:37.088 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-03-11T22:50:37.088 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:17.2.7-620.gc186d0ca.el9.x86_64 25/26 2024-03-11T22:50:37.634 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:28 ago on Mon 11 Mar 2024 10:50:09 PM UTC. 2024-03-11T22:50:37.786 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-11T22:50:37.787 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-11T22:50:37.787 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-03-11T22:50:37.787 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-11T22:50:37.787 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-11T22:50:37.787 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-test x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 48 M 2024-03-11T22:50:37.788 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-03-11T22:50:37.788 INFO:teuthology.orchestra.run.smithi163.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-11T22:50:37.788 INFO:teuthology.orchestra.run.smithi163.stdout: libcephsqlite x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 162 k 2024-03-11T22:50:37.788 INFO:teuthology.orchestra.run.smithi163.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-11T22:50:37.788 INFO:teuthology.orchestra.run.smithi163.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-11T22:50:37.788 INFO:teuthology.orchestra.run.smithi163.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-11T22:50:37.788 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:50:37.788 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-11T22:50:37.788 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-11T22:50:37.788 INFO:teuthology.orchestra.run.smithi163.stdout:Install 6 Packages 2024-03-11T22:50:37.788 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:50:37.789 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 49 M 2024-03-11T22:50:37.789 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 196 M 2024-03-11T22:50:37.789 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-11T22:50:38.074 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-03-11T22:50:38.074 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:50:38.074 INFO:teuthology.orchestra.run.smithi153.stdout:Upgraded: 2024-03-11T22:50:38.074 INFO:teuthology.orchestra.run.smithi153.stdout: librados2-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:50:38.075 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:50:38.075 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-11T22:50:38.075 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:50:38.075 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:50:38.075 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:50:38.075 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:50:38.075 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-11T22:50:38.075 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:50:38.075 INFO:teuthology.orchestra.run.smithi153.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-11T22:50:38.075 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-11T22:50:38.075 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-11T22:50:38.075 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:50:38.075 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-11T22:50:38.076 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:50:38.076 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-11T22:50:38.076 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-11T22:50:38.076 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:50:38.076 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:50:38.076 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:50:38.076 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-11T22:50:38.076 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:50:38.076 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:50:38.076 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:50:38.076 INFO:teuthology.orchestra.run.smithi153.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-11T22:50:38.076 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:50:38.076 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-11T22:50:38.305 INFO:teuthology.orchestra.run.smithi163.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.0 MB/s | 217 kB 00:00 2024-03-11T22:50:38.364 INFO:teuthology.orchestra.run.smithi163.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 300 kB 00:00 2024-03-11T22:50:38.406 INFO:teuthology.orchestra.run.smithi163.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-11T22:50:38.458 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-test 2024-03-11T22:50:38.698 INFO:teuthology.orchestra.run.smithi163.stdout:(4/6): libcephsqlite-17.2.7-620.gc186d0ca.el9.x 272 kB/s | 162 kB 00:00 2024-03-11T22:50:38.933 INFO:teuthology.orchestra.run.smithi163.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 355 kB/s | 187 kB 00:00 2024-03-11T22:50:39.027 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:29 ago on Mon 11 Mar 2024 10:50:10 PM UTC. 2024-03-11T22:50:39.182 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-11T22:50:39.183 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-11T22:50:39.183 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-11T22:50:39.183 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-11T22:50:39.183 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-11T22:50:39.183 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 48 M 2024-03-11T22:50:39.183 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-11T22:50:39.183 INFO:teuthology.orchestra.run.smithi153.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-11T22:50:39.183 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 162 k 2024-03-11T22:50:39.183 INFO:teuthology.orchestra.run.smithi153.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-11T22:50:39.183 INFO:teuthology.orchestra.run.smithi153.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-11T22:50:39.183 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-11T22:50:39.183 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:50:39.184 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-11T22:50:39.184 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-11T22:50:39.184 INFO:teuthology.orchestra.run.smithi153.stdout:Install 6 Packages 2024-03-11T22:50:39.184 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:50:39.184 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 49 M 2024-03-11T22:50:39.184 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 196 M 2024-03-11T22:50:39.184 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-11T22:50:39.417 INFO:teuthology.orchestra.run.smithi153.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-03-11T22:50:39.475 INFO:teuthology.orchestra.run.smithi153.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 300 kB 00:00 2024-03-11T22:50:39.509 INFO:teuthology.orchestra.run.smithi153.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-11T22:50:39.626 INFO:teuthology.orchestra.run.smithi153.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-11T22:50:39.876 INFO:teuthology.orchestra.run.smithi153.stdout:(5/6): libcephsqlite-17.2.7-620.gc186d0ca.el9.x 250 kB/s | 162 kB 00:00 2024-03-11T22:50:40.859 INFO:teuthology.orchestra.run.smithi163.stdout:(6/6): ceph-test-17.2.7-620.gc186d0ca.el9.x86_6 17 MB/s | 48 MB 00:02 2024-03-11T22:50:40.862 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:50:40.862 INFO:teuthology.orchestra.run.smithi163.stdout:Total 16 MB/s | 49 MB 00:03 2024-03-11T22:50:40.877 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-11T22:50:40.897 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-11T22:50:40.898 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-11T22:50:41.297 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-11T22:50:41.297 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-11T22:50:41.882 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-11T22:50:41.968 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-11T22:50:42.001 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-11T22:50:42.033 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-11T22:50:42.075 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-11T22:50:42.103 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephsqlite-2:17.2.7-620.gc186d0ca.el9.x86_64 5/6 2024-03-11T22:50:43.809 INFO:teuthology.orchestra.run.smithi153.stdout:(6/6): ceph-test-17.2.7-620.gc186d0ca.el9.x86_6 10 MB/s | 48 MB 00:04 2024-03-11T22:50:43.812 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:50:43.813 INFO:teuthology.orchestra.run.smithi153.stdout:Total 11 MB/s | 49 MB 00:04 2024-03-11T22:50:43.827 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-11T22:50:43.848 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-11T22:50:43.848 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-11T22:50:44.250 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-11T22:50:44.250 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-11T22:50:44.839 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-11T22:50:44.910 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-11T22:50:44.952 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-11T22:50:44.984 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-11T22:50:45.026 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-11T22:50:45.054 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephsqlite-2:17.2.7-620.gc186d0ca.el9.x86_64 5/6 2024-03-11T22:50:46.551 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephsqlite-2:17.2.7-620.gc186d0ca.el9.x86_64 5/6 2024-03-11T22:50:46.636 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-test-2:17.2.7-620.gc186d0ca.el9.x86_64 6/6 2024-03-11T22:50:49.038 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-test-2:17.2.7-620.gc186d0ca.el9.x86_64 6/6 2024-03-11T22:50:49.038 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-test-2:17.2.7-620.gc186d0ca.el9.x86_64 1/6 2024-03-11T22:50:49.038 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephsqlite-2:17.2.7-620.gc186d0ca.el9.x86_64 2/6 2024-03-11T22:50:49.038 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-11T22:50:49.039 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-11T22:50:49.039 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-11T22:50:49.405 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-11T22:50:49.405 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:50:49.405 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-11T22:50:49.405 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-test-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:50:49.405 INFO:teuthology.orchestra.run.smithi163.stdout: jq-1.6-16.el9.x86_64 2024-03-11T22:50:49.405 INFO:teuthology.orchestra.run.smithi163.stdout: libcephsqlite-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:50:49.405 INFO:teuthology.orchestra.run.smithi163.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-11T22:50:49.405 INFO:teuthology.orchestra.run.smithi163.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-11T22:50:49.405 INFO:teuthology.orchestra.run.smithi163.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-11T22:50:49.406 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:50:49.406 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-11T22:50:49.555 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephsqlite-2:17.2.7-620.gc186d0ca.el9.x86_64 5/6 2024-03-11T22:50:49.633 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-test-2:17.2.7-620.gc186d0ca.el9.x86_64 6/6 2024-03-11T22:50:49.686 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph 2024-03-11T22:50:50.257 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:41 ago on Mon 11 Mar 2024 10:50:09 PM UTC. 2024-03-11T22:50:50.411 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-11T22:50:50.413 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================== 2024-03-11T22:50:50.413 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-03-11T22:50:50.413 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================== 2024-03-11T22:50:50.413 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-11T22:50:50.414 INFO:teuthology.orchestra.run.smithi163.stdout: ceph x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 6.5 k 2024-03-11T22:50:50.414 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-03-11T22:50:50.414 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mds x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 2.1 M 2024-03-11T22:50:50.414 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 1.4 M 2024-03-11T22:50:50.414 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-modules-core noarch 2:17.2.7-620.gc186d0ca.el9 ceph-noarch 239 k 2024-03-11T22:50:50.414 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mon x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 3.7 M 2024-03-11T22:50:50.414 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-osd x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 15 M 2024-03-11T22:50:50.414 INFO:teuthology.orchestra.run.smithi163.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-11T22:50:50.414 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-11T22:50:50.414 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-11T22:50:50.414 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-11T22:50:50.414 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-11T22:50:50.414 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-11T22:50:50.414 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-11T22:50:50.415 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-11T22:50:50.415 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-11T22:50:50.415 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-11T22:50:50.415 INFO:teuthology.orchestra.run.smithi163.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-11T22:50:50.415 INFO:teuthology.orchestra.run.smithi163.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-11T22:50:50.415 INFO:teuthology.orchestra.run.smithi163.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-11T22:50:50.415 INFO:teuthology.orchestra.run.smithi163.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-11T22:50:50.415 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-11T22:50:50.415 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-11T22:50:50.415 INFO:teuthology.orchestra.run.smithi163.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-11T22:50:50.415 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-11T22:50:50.415 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-11T22:50:50.415 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-11T22:50:50.415 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-11T22:50:50.416 INFO:teuthology.orchestra.run.smithi163.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-11T22:50:50.416 INFO:teuthology.orchestra.run.smithi163.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-11T22:50:50.416 INFO:teuthology.orchestra.run.smithi163.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-11T22:50:50.416 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-11T22:50:50.416 INFO:teuthology.orchestra.run.smithi163.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-11T22:50:50.416 INFO:teuthology.orchestra.run.smithi163.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-11T22:50:50.416 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:50:50.416 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-11T22:50:50.416 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================== 2024-03-11T22:50:50.416 INFO:teuthology.orchestra.run.smithi163.stdout:Install 33 Packages 2024-03-11T22:50:50.416 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:50:50.417 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 27 M 2024-03-11T22:50:50.417 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 99 M 2024-03-11T22:50:50.417 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-11T22:50:50.996 INFO:teuthology.orchestra.run.smithi163.stdout:(1/33): ceph-17.2.7-620.gc186d0ca.el9.x86_64.rp 27 kB/s | 6.5 kB 00:00 2024-03-11T22:50:51.129 INFO:teuthology.orchestra.run.smithi163.stdout:(2/33): ceph-mds-17.2.7-620.gc186d0ca.el9.x86_6 5.5 MB/s | 2.1 MB 00:00 2024-03-11T22:50:51.430 INFO:teuthology.orchestra.run.smithi163.stdout:(3/33): ceph-mgr-17.2.7-620.gc186d0ca.el9.x86_6 2.1 MB/s | 1.4 MB 00:00 2024-03-11T22:50:51.821 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-test-2:17.2.7-620.gc186d0ca.el9.x86_64 6/6 2024-03-11T22:50:51.822 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-test-2:17.2.7-620.gc186d0ca.el9.x86_64 1/6 2024-03-11T22:50:51.822 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephsqlite-2:17.2.7-620.gc186d0ca.el9.x86_64 2/6 2024-03-11T22:50:51.822 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-11T22:50:51.822 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-11T22:50:51.822 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-11T22:50:51.938 INFO:teuthology.orchestra.run.smithi163.stdout:(4/33): ceph-osd-17.2.7-620.gc186d0ca.el9.x86_6 19 MB/s | 15 MB 00:00 2024-03-11T22:50:51.964 INFO:teuthology.orchestra.run.smithi163.stdout:(5/33): ceph-mgr-modules-core-17.2.7-620.gc186d 448 kB/s | 239 kB 00:00 2024-03-11T22:50:52.089 INFO:teuthology.orchestra.run.smithi163.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 314 kB/s | 32 kB 00:00 2024-03-11T22:50:52.114 INFO:teuthology.orchestra.run.smithi163.stdout:(7/33): python3-mako-1.1.4-6.el9.noarch.rpm 822 kB/s | 144 kB 00:00 2024-03-11T22:50:52.131 INFO:teuthology.orchestra.run.smithi163.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-11T22:50:52.156 INFO:teuthology.orchestra.run.smithi163.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 929 kB/s | 39 kB 00:00 2024-03-11T22:50:52.190 INFO:teuthology.orchestra.run.smithi163.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-11T22:50:52.225 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-11T22:50:52.225 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:50:52.225 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-11T22:50:52.225 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:50:52.225 INFO:teuthology.orchestra.run.smithi153.stdout: jq-1.6-16.el9.x86_64 2024-03-11T22:50:52.225 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:50:52.225 INFO:teuthology.orchestra.run.smithi153.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-11T22:50:52.226 INFO:teuthology.orchestra.run.smithi153.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-11T22:50:52.226 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-11T22:50:52.226 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:50:52.226 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-11T22:50:52.257 INFO:teuthology.orchestra.run.smithi163.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2024-03-11T22:50:52.282 INFO:teuthology.orchestra.run.smithi163.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-11T22:50:52.308 INFO:teuthology.orchestra.run.smithi163.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-11T22:50:52.333 INFO:teuthology.orchestra.run.smithi163.stdout:(14/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-11T22:50:52.442 INFO:teuthology.orchestra.run.smithi163.stdout:(15/33): python3-requests-2.25.1-8.el9.noarch.r 937 kB/s | 125 kB 00:00 2024-03-11T22:50:52.534 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph 2024-03-11T22:50:52.675 INFO:teuthology.orchestra.run.smithi163.stdout:(16/33): ceph-mon-17.2.7-620.gc186d0ca.el9.x86_ 2.2 MB/s | 3.7 MB 00:01 2024-03-11T22:50:52.701 INFO:teuthology.orchestra.run.smithi163.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 118 kB/s | 43 kB 00:00 2024-03-11T22:50:52.876 INFO:teuthology.orchestra.run.smithi163.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 61 kB/s | 11 kB 00:00 2024-03-11T22:50:52.910 INFO:teuthology.orchestra.run.smithi163.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 367 kB/s | 172 kB 00:00 2024-03-11T22:50:52.935 INFO:teuthology.orchestra.run.smithi163.stdout:(20/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-03-11T22:50:52.977 INFO:teuthology.orchestra.run.smithi163.stdout:(21/33): python3-jaraco-classes-3.2.1-5.el9.noa 175 kB/s | 18 kB 00:00 2024-03-11T22:50:53.012 INFO:teuthology.orchestra.run.smithi163.stdout:(22/33): python3-jaraco-collections-3.0.0-8.el9 230 kB/s | 23 kB 00:00 2024-03-11T22:50:53.036 INFO:teuthology.orchestra.run.smithi163.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 193 kB/s | 19 kB 00:00 2024-03-11T22:50:53.079 INFO:teuthology.orchestra.run.smithi163.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 195 kB/s | 20 kB 00:00 2024-03-11T22:50:53.110 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:43 ago on Mon 11 Mar 2024 10:50:10 PM UTC. 2024-03-11T22:50:53.112 INFO:teuthology.orchestra.run.smithi163.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 460 kB/s | 46 kB 00:00 2024-03-11T22:50:53.140 INFO:teuthology.orchestra.run.smithi163.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 784 kB/s | 79 kB 00:00 2024-03-11T22:50:53.212 INFO:teuthology.orchestra.run.smithi163.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 164 kB/s | 16 kB 00:00 2024-03-11T22:50:53.246 INFO:teuthology.orchestra.run.smithi163.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 829 kB/s | 90 kB 00:00 2024-03-11T22:50:53.266 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-11T22:50:53.269 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================== 2024-03-11T22:50:53.269 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-11T22:50:53.269 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================== 2024-03-11T22:50:53.270 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-11T22:50:53.270 INFO:teuthology.orchestra.run.smithi153.stdout: ceph x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 6.5 k 2024-03-11T22:50:53.270 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-11T22:50:53.270 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 2.1 M 2024-03-11T22:50:53.270 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 1.4 M 2024-03-11T22:50:53.270 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core noarch 2:17.2.7-620.gc186d0ca.el9 ceph-noarch 239 k 2024-03-11T22:50:53.270 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 3.7 M 2024-03-11T22:50:53.270 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 15 M 2024-03-11T22:50:53.271 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-11T22:50:53.271 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-11T22:50:53.271 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-11T22:50:53.271 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-11T22:50:53.271 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-11T22:50:53.271 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-11T22:50:53.271 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-11T22:50:53.272 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-11T22:50:53.272 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-11T22:50:53.272 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-11T22:50:53.272 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-11T22:50:53.272 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-11T22:50:53.272 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-11T22:50:53.272 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-11T22:50:53.272 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-11T22:50:53.272 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-11T22:50:53.272 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-11T22:50:53.273 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-11T22:50:53.273 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-11T22:50:53.273 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-11T22:50:53.273 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-11T22:50:53.273 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-11T22:50:53.273 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-11T22:50:53.273 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-11T22:50:53.273 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-11T22:50:53.273 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-11T22:50:53.274 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-11T22:50:53.274 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:50:53.274 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-11T22:50:53.274 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================== 2024-03-11T22:50:53.274 INFO:teuthology.orchestra.run.smithi153.stdout:Install 33 Packages 2024-03-11T22:50:53.274 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:50:53.275 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 27 M 2024-03-11T22:50:53.275 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 99 M 2024-03-11T22:50:53.275 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-11T22:50:53.305 INFO:teuthology.orchestra.run.smithi163.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 389 kB/s | 36 kB 00:00 2024-03-11T22:50:53.371 INFO:teuthology.orchestra.run.smithi163.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-03-11T22:50:53.422 INFO:teuthology.orchestra.run.smithi163.stdout:(31/33): python3-pecan-1.4.2-3.el9.noarch.rpm 793 kB/s | 272 kB 00:00 2024-03-11T22:50:53.472 INFO:teuthology.orchestra.run.smithi163.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 199 kB/s | 20 kB 00:00 2024-03-11T22:50:53.505 INFO:teuthology.orchestra.run.smithi163.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.1 MB/s | 427 kB 00:00 2024-03-11T22:50:53.508 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:50:53.508 INFO:teuthology.orchestra.run.smithi163.stdout:Total 8.6 MB/s | 27 MB 00:03 2024-03-11T22:50:53.522 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-11T22:50:53.563 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-11T22:50:53.563 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-11T22:50:53.688 INFO:teuthology.orchestra.run.smithi153.stdout:(1/33): ceph-17.2.7-620.gc186d0ca.el9.x86_64.rp 40 kB/s | 6.5 kB 00:00 2024-03-11T22:50:53.918 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-11T22:50:53.919 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-11T22:50:54.021 INFO:teuthology.orchestra.run.smithi153.stdout:(2/33): ceph-mon-17.2.7-620.gc186d0ca.el9.x86_6 11 MB/s | 3.7 MB 00:00 2024-03-11T22:50:54.197 INFO:teuthology.orchestra.run.smithi153.stdout:(3/33): ceph-mgr-17.2.7-620.gc186d0ca.el9.x86_6 2.1 MB/s | 1.4 MB 00:00 2024-03-11T22:50:54.247 INFO:teuthology.orchestra.run.smithi153.stdout:(4/33): ceph-mds-17.2.7-620.gc186d0ca.el9.x86_6 2.9 MB/s | 2.1 MB 00:00 2024-03-11T22:50:54.314 INFO:teuthology.orchestra.run.smithi153.stdout:(5/33): ceph-mgr-modules-core-17.2.7-620.gc186d 2.0 MB/s | 239 kB 00:00 2024-03-11T22:50:54.397 INFO:teuthology.orchestra.run.smithi153.stdout:(6/33): python3-mako-1.1.4-6.el9.noarch.rpm 961 kB/s | 144 kB 00:00 2024-03-11T22:50:54.422 INFO:teuthology.orchestra.run.smithi153.stdout:(7/33): python3-markupsafe-1.1.1-12.el9.x86_64. 294 kB/s | 32 kB 00:00 2024-03-11T22:50:54.448 INFO:teuthology.orchestra.run.smithi153.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 943 kB/s | 47 kB 00:00 2024-03-11T22:50:54.473 INFO:teuthology.orchestra.run.smithi153.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 769 kB/s | 39 kB 00:00 2024-03-11T22:50:54.515 INFO:teuthology.orchestra.run.smithi153.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-11T22:50:54.574 INFO:teuthology.orchestra.run.smithi153.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2024-03-11T22:50:54.599 INFO:teuthology.orchestra.run.smithi153.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-11T22:50:54.625 INFO:teuthology.orchestra.run.smithi153.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-11T22:50:54.650 INFO:teuthology.orchestra.run.smithi153.stdout:(14/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-11T22:50:54.808 INFO:teuthology.orchestra.run.smithi153.stdout:(15/33): ceph-osd-17.2.7-620.gc186d0ca.el9.x86_ 19 MB/s | 15 MB 00:00 2024-03-11T22:50:54.917 INFO:teuthology.orchestra.run.smithi153.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 163 kB/s | 43 kB 00:00 2024-03-11T22:50:55.034 INFO:teuthology.orchestra.run.smithi153.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 763 kB/s | 172 kB 00:00 2024-03-11T22:50:55.084 INFO:teuthology.orchestra.run.smithi153.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 215 kB/s | 11 kB 00:00 2024-03-11T22:50:55.118 INFO:teuthology.orchestra.run.smithi153.stdout:(19/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2024-03-11T22:50:55.134 INFO:teuthology.orchestra.run.smithi153.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 358 kB/s | 18 kB 00:00 2024-03-11T22:50:55.168 INFO:teuthology.orchestra.run.smithi153.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2024-03-11T22:50:55.185 INFO:teuthology.orchestra.run.smithi153.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-03-11T22:50:55.186 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-11T22:50:55.210 INFO:teuthology.orchestra.run.smithi153.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 471 kB/s | 20 kB 00:00 2024-03-11T22:50:55.235 INFO:teuthology.orchestra.run.smithi153.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 927 kB/s | 46 kB 00:00 2024-03-11T22:50:55.258 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-03-11T22:50:55.269 INFO:teuthology.orchestra.run.smithi153.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-03-11T22:50:55.291 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-03-11T22:50:55.319 INFO:teuthology.orchestra.run.smithi153.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 330 kB/s | 16 kB 00:00 2024-03-11T22:50:55.324 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-03-11T22:50:55.345 INFO:teuthology.orchestra.run.smithi153.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-03-11T22:50:55.378 INFO:teuthology.orchestra.run.smithi153.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-03-11T22:50:55.404 INFO:teuthology.orchestra.run.smithi153.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 610 kB/s | 36 kB 00:00 2024-03-11T22:50:55.415 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-03-11T22:50:55.454 INFO:teuthology.orchestra.run.smithi153.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-11T22:50:55.487 INFO:teuthology.orchestra.run.smithi153.stdout:(31/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2024-03-11T22:50:55.502 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-03-11T22:50:55.513 INFO:teuthology.orchestra.run.smithi153.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 341 kB/s | 20 kB 00:00 2024-03-11T22:50:55.543 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-03-11T22:50:55.571 INFO:teuthology.orchestra.run.smithi153.stdout:(33/33): python3-requests-2.25.1-8.el9.noarch.r 132 kB/s | 125 kB 00:00 2024-03-11T22:50:55.576 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:50:55.576 INFO:teuthology.orchestra.run.smithi153.stdout:Total 12 MB/s | 27 MB 00:02 2024-03-11T22:50:55.589 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-11T22:50:55.624 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-03-11T22:50:55.628 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-11T22:50:55.628 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-11T22:50:55.677 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-03-11T22:50:55.716 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-03-11T22:50:55.764 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-03-11T22:50:55.820 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-03-11T22:50:55.982 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-11T22:50:55.982 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-11T22:50:56.006 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-03-11T22:50:56.062 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-03-11T22:50:56.101 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-03-11T22:50:56.135 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-03-11T22:50:56.165 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-03-11T22:50:56.258 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-03-11T22:50:56.293 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-11T22:50:56.326 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-03-11T22:50:56.366 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-03-11T22:50:56.401 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-03-11T22:50:56.436 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-03-11T22:50:56.526 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-03-11T22:50:56.568 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-03-11T22:50:56.639 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-03-11T22:50:56.777 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-03-11T22:50:56.902 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-11T22:50:57.050 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-620.gc186d0ca.el9.n 28/33 2024-03-11T22:50:57.073 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-2:17.2.7-620.gc186d0ca.el9.x86_64 29/33 2024-03-11T22:50:57.097 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-2:17.2.7-620.gc186d0ca.el9.x86_64 29/33 2024-03-11T22:50:57.098 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:50:57.098 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-11T22:50:57.098 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T22:50:57.098 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T22:50:57.098 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:50:57.220 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-11T22:50:57.274 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-03-11T22:50:57.306 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-03-11T22:50:57.348 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-03-11T22:50:57.440 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-03-11T22:50:57.526 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-03-11T22:50:57.567 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-03-11T22:50:57.647 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-03-11T22:50:57.701 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-03-11T22:50:57.739 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-03-11T22:50:57.788 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-03-11T22:50:57.843 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-03-11T22:50:58.076 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-03-11T22:50:58.118 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-03-11T22:50:58.157 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-03-11T22:50:58.200 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-03-11T22:50:58.230 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-03-11T22:50:58.307 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-03-11T22:50:58.340 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-11T22:50:58.383 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-03-11T22:50:58.421 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-03-11T22:50:58.449 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-osd-2:17.2.7-620.gc186d0ca.el9.x86_64 30/33 2024-03-11T22:50:58.457 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-03-11T22:50:58.476 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-osd-2:17.2.7-620.gc186d0ca.el9.x86_64 30/33 2024-03-11T22:50:58.476 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:50:58.476 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-11T22:50:58.476 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T22:50:58.476 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T22:50:58.477 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:50:58.491 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-03-11T22:50:58.582 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-03-11T22:50:58.631 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-03-11T22:50:58.704 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-03-11T22:50:58.840 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-03-11T22:50:58.875 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mon-2:17.2.7-620.gc186d0ca.el9.x86_64 31/33 2024-03-11T22:50:58.899 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mon-2:17.2.7-620.gc186d0ca.el9.x86_64 31/33 2024-03-11T22:50:58.899 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:50:58.899 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-11T22:50:58.899 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T22:50:58.899 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T22:50:58.899 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:50:58.966 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-11T22:50:59.101 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mds-2:17.2.7-620.gc186d0ca.el9.x86_64 32/33 2024-03-11T22:50:59.120 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-620.gc186d0ca.el9.n 28/33 2024-03-11T22:50:59.124 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mds-2:17.2.7-620.gc186d0ca.el9.x86_64 32/33 2024-03-11T22:50:59.124 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:50:59.124 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-11T22:50:59.124 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T22:50:59.124 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T22:50:59.124 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:50:59.152 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-2:17.2.7-620.gc186d0ca.el9.x86_64 29/33 2024-03-11T22:50:59.176 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-2:17.2.7-620.gc186d0ca.el9.x86_64 29/33 2024-03-11T22:50:59.177 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:50:59.177 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-11T22:50:59.177 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T22:50:59.177 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T22:50:59.177 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:50:59.198 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-2:17.2.7-620.gc186d0ca.el9.x86_64 33/33 2024-03-11T22:51:00.528 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-osd-2:17.2.7-620.gc186d0ca.el9.x86_64 30/33 2024-03-11T22:51:00.555 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-osd-2:17.2.7-620.gc186d0ca.el9.x86_64 30/33 2024-03-11T22:51:00.555 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:51:00.555 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-11T22:51:00.555 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T22:51:00.555 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T22:51:00.556 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:00.920 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-2:17.2.7-620.gc186d0ca.el9.x86_64 33/33 2024-03-11T22:51:00.920 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-2:17.2.7-620.gc186d0ca.el9.x86_64 1/33 2024-03-11T22:51:00.920 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mds-2:17.2.7-620.gc186d0ca.el9.x86_64 2/33 2024-03-11T22:51:00.921 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-2:17.2.7-620.gc186d0ca.el9.x86_64 3/33 2024-03-11T22:51:00.921 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mon-2:17.2.7-620.gc186d0ca.el9.x86_64 4/33 2024-03-11T22:51:00.921 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-osd-2:17.2.7-620.gc186d0ca.el9.x86_64 5/33 2024-03-11T22:51:00.921 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-620.gc186d0ca.el9.n 6/33 2024-03-11T22:51:00.921 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-03-11T22:51:00.921 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-03-11T22:51:00.921 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-03-11T22:51:00.921 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-03-11T22:51:00.921 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-03-11T22:51:00.921 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-03-11T22:51:00.921 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-03-11T22:51:00.921 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-03-11T22:51:00.921 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-03-11T22:51:00.922 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-03-11T22:51:00.922 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-03-11T22:51:00.922 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-11T22:51:00.922 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-03-11T22:51:00.922 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-03-11T22:51:00.922 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-03-11T22:51:00.922 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-03-11T22:51:00.922 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-03-11T22:51:00.922 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-03-11T22:51:00.922 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-03-11T22:51:00.922 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-03-11T22:51:00.922 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-11T22:51:00.922 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-03-11T22:51:00.922 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-03-11T22:51:00.923 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-03-11T22:51:00.923 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-03-11T22:51:00.923 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-03-11T22:51:00.928 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mon-2:17.2.7-620.gc186d0ca.el9.x86_64 31/33 2024-03-11T22:51:00.952 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mon-2:17.2.7-620.gc186d0ca.el9.x86_64 31/33 2024-03-11T22:51:00.952 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:51:00.952 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-11T22:51:00.952 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T22:51:00.952 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T22:51:00.952 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:01.154 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mds-2:17.2.7-620.gc186d0ca.el9.x86_64 32/33 2024-03-11T22:51:01.177 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mds-2:17.2.7-620.gc186d0ca.el9.x86_64 32/33 2024-03-11T22:51:01.177 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:51:01.178 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-11T22:51:01.178 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T22:51:01.178 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T22:51:01.178 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:01.244 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-2:17.2.7-620.gc186d0ca.el9.x86_64 33/33 2024-03-11T22:51:01.879 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-03-11T22:51:01.880 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:01.880 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-11T22:51:01.880 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:51:01.880 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mds-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:51:01.880 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:51:01.880 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-modules-core-2:17.2.7-620.gc186d0ca.el9.noarch 2024-03-11T22:51:01.880 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mon-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:51:01.880 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-osd-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:51:01.880 INFO:teuthology.orchestra.run.smithi163.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-11T22:51:01.880 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-11T22:51:01.880 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-11T22:51:01.881 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-11T22:51:01.881 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-11T22:51:01.881 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-11T22:51:01.881 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-11T22:51:01.881 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-11T22:51:01.881 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-11T22:51:01.881 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-11T22:51:01.881 INFO:teuthology.orchestra.run.smithi163.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-11T22:51:01.881 INFO:teuthology.orchestra.run.smithi163.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-11T22:51:01.881 INFO:teuthology.orchestra.run.smithi163.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-11T22:51:01.881 INFO:teuthology.orchestra.run.smithi163.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-11T22:51:01.881 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-11T22:51:01.881 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ply-3.11-14.el9.noarch 2024-03-11T22:51:01.881 INFO:teuthology.orchestra.run.smithi163.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-11T22:51:01.882 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-11T22:51:01.882 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-11T22:51:01.882 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-11T22:51:01.882 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-11T22:51:01.882 INFO:teuthology.orchestra.run.smithi163.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-11T22:51:01.882 INFO:teuthology.orchestra.run.smithi163.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-11T22:51:01.882 INFO:teuthology.orchestra.run.smithi163.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-11T22:51:01.882 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-11T22:51:01.882 INFO:teuthology.orchestra.run.smithi163.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-11T22:51:01.882 INFO:teuthology.orchestra.run.smithi163.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-11T22:51:01.882 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:01.882 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-11T22:51:02.234 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-base 2024-03-11T22:51:02.794 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:53 ago on Mon 11 Mar 2024 10:50:09 PM UTC. 2024-03-11T22:51:02.892 INFO:teuthology.orchestra.run.smithi163.stdout:Package ceph-base-2:17.2.7-620.gc186d0ca.el9.x86_64 is already installed. 2024-03-11T22:51:02.922 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-2:17.2.7-620.gc186d0ca.el9.x86_64 33/33 2024-03-11T22:51:02.923 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-2:17.2.7-620.gc186d0ca.el9.x86_64 1/33 2024-03-11T22:51:02.923 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mds-2:17.2.7-620.gc186d0ca.el9.x86_64 2/33 2024-03-11T22:51:02.923 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-2:17.2.7-620.gc186d0ca.el9.x86_64 3/33 2024-03-11T22:51:02.923 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mon-2:17.2.7-620.gc186d0ca.el9.x86_64 4/33 2024-03-11T22:51:02.923 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-osd-2:17.2.7-620.gc186d0ca.el9.x86_64 5/33 2024-03-11T22:51:02.923 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-620.gc186d0ca.el9.n 6/33 2024-03-11T22:51:02.923 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-03-11T22:51:02.923 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-03-11T22:51:02.923 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-03-11T22:51:02.923 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-03-11T22:51:02.923 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-03-11T22:51:02.923 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-03-11T22:51:02.923 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-03-11T22:51:02.924 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-03-11T22:51:02.924 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-03-11T22:51:02.924 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-03-11T22:51:02.924 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-03-11T22:51:02.924 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-11T22:51:02.924 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-03-11T22:51:02.924 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-03-11T22:51:02.924 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-03-11T22:51:02.924 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-03-11T22:51:02.924 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-03-11T22:51:02.924 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-03-11T22:51:02.924 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-03-11T22:51:02.925 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-03-11T22:51:02.925 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-11T22:51:02.925 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-03-11T22:51:02.926 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-03-11T22:51:02.926 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-03-11T22:51:02.926 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-03-11T22:51:02.926 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-03-11T22:51:02.944 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-11T22:51:02.945 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-03-11T22:51:02.945 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-11T22:51:03.010 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install cephadm 2024-03-11T22:51:03.569 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:54 ago on Mon 11 Mar 2024 10:50:09 PM UTC. 2024-03-11T22:51:03.721 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-11T22:51:03.722 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-11T22:51:03.722 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-03-11T22:51:03.722 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-11T22:51:03.722 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-11T22:51:03.722 INFO:teuthology.orchestra.run.smithi163.stdout: cephadm noarch 2:17.2.7-620.gc186d0ca.el9 ceph-noarch 82 k 2024-03-11T22:51:03.722 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:03.722 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-11T22:51:03.722 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-11T22:51:03.722 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-11T22:51:03.723 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:03.723 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 82 k 2024-03-11T22:51:03.723 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 369 k 2024-03-11T22:51:03.723 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-11T22:51:03.723 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-03-11T22:51:03.723 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:03.723 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-11T22:51:03.723 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:51:03.724 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:51:03.724 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:51:03.724 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core-2:17.2.7-620.gc186d0ca.el9.noarch 2024-03-11T22:51:03.724 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:51:03.724 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:51:03.724 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-11T22:51:03.724 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-11T22:51:03.724 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-11T22:51:03.724 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-11T22:51:03.724 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-11T22:51:03.724 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-11T22:51:03.724 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-11T22:51:03.724 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-11T22:51:03.724 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-11T22:51:03.724 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-11T22:51:03.725 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-11T22:51:03.725 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-11T22:51:03.725 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-11T22:51:03.725 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-11T22:51:03.725 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-11T22:51:03.725 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply-3.11-14.el9.noarch 2024-03-11T22:51:03.725 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-11T22:51:03.725 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-11T22:51:03.725 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-11T22:51:03.725 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-11T22:51:03.725 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-11T22:51:03.725 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-11T22:51:03.725 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-11T22:51:03.725 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-11T22:51:03.726 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-11T22:51:03.726 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-11T22:51:03.726 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-11T22:51:03.726 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:03.726 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-11T22:51:04.044 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-base 2024-03-11T22:51:04.085 INFO:teuthology.orchestra.run.smithi163.stdout:cephadm-17.2.7-620.gc186d0ca.el9.noarch.rpm 225 kB/s | 82 kB 00:00 2024-03-11T22:51:04.085 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:51:04.085 INFO:teuthology.orchestra.run.smithi163.stdout:Total 224 kB/s | 82 kB 00:00 2024-03-11T22:51:04.086 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-11T22:51:04.091 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-11T22:51:04.091 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-11T22:51:04.092 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-11T22:51:04.093 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-11T22:51:04.170 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-11T22:51:04.574 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: cephadm-2:17.2.7-620.gc186d0ca.el9.noarch 1/1 2024-03-11T22:51:04.605 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:54 ago on Mon 11 Mar 2024 10:50:10 PM UTC. 2024-03-11T22:51:04.649 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : cephadm-2:17.2.7-620.gc186d0ca.el9.noarch 1/1 2024-03-11T22:51:04.702 INFO:teuthology.orchestra.run.smithi153.stdout:Package ceph-base-2:17.2.7-620.gc186d0ca.el9.x86_64 is already installed. 2024-03-11T22:51:04.754 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-11T22:51:04.755 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-11T22:51:04.755 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-11T22:51:04.819 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install cephadm 2024-03-11T22:51:05.062 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: cephadm-2:17.2.7-620.gc186d0ca.el9.noarch 1/1 2024-03-11T22:51:05.378 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:55 ago on Mon 11 Mar 2024 10:50:10 PM UTC. 2024-03-11T22:51:05.381 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : cephadm-2:17.2.7-620.gc186d0ca.el9.noarch 1/1 2024-03-11T22:51:05.382 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:05.382 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-11T22:51:05.382 INFO:teuthology.orchestra.run.smithi163.stdout: cephadm-2:17.2.7-620.gc186d0ca.el9.noarch 2024-03-11T22:51:05.382 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:05.382 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-11T22:51:05.529 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-11T22:51:05.529 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-11T22:51:05.530 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-11T22:51:05.530 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-11T22:51:05.530 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-11T22:51:05.530 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm noarch 2:17.2.7-620.gc186d0ca.el9 ceph-noarch 82 k 2024-03-11T22:51:05.530 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:05.530 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-11T22:51:05.530 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-11T22:51:05.530 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-11T22:51:05.530 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:05.530 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 82 k 2024-03-11T22:51:05.530 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 369 k 2024-03-11T22:51:05.530 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-11T22:51:05.585 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-immutable-object-cache 2024-03-11T22:51:05.744 INFO:teuthology.orchestra.run.smithi153.stdout:cephadm-17.2.7-620.gc186d0ca.el9.noarch.rpm 382 kB/s | 82 kB 00:00 2024-03-11T22:51:05.745 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:51:05.745 INFO:teuthology.orchestra.run.smithi153.stdout:Total 380 kB/s | 82 kB 00:00 2024-03-11T22:51:05.745 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-11T22:51:05.750 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-11T22:51:05.750 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-11T22:51:05.751 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-11T22:51:05.752 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-11T22:51:05.816 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-11T22:51:06.152 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:57 ago on Mon 11 Mar 2024 10:50:09 PM UTC. 2024-03-11T22:51:06.182 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:17.2.7-620.gc186d0ca.el9.noarch 1/1 2024-03-11T22:51:06.291 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephadm-2:17.2.7-620.gc186d0ca.el9.noarch 1/1 2024-03-11T22:51:06.305 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-11T22:51:06.305 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-11T22:51:06.305 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-03-11T22:51:06.305 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-11T22:51:06.305 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-11T22:51:06.305 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 137 k 2024-03-11T22:51:06.305 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:06.305 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-11T22:51:06.306 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-11T22:51:06.306 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-11T22:51:06.306 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:06.306 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 137 k 2024-03-11T22:51:06.306 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 379 k 2024-03-11T22:51:06.306 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-11T22:51:06.713 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:17.2.7-620.gc186d0ca.el9.noarch 1/1 2024-03-11T22:51:06.817 INFO:teuthology.orchestra.run.smithi163.stdout:ceph-immutable-object-cache-17.2.7-620.gc186d0c 268 kB/s | 137 kB 00:00 2024-03-11T22:51:06.817 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:51:06.817 INFO:teuthology.orchestra.run.smithi163.stdout:Total 268 kB/s | 137 kB 00:00 2024-03-11T22:51:06.817 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-11T22:51:06.825 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-11T22:51:06.825 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-11T22:51:06.877 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-11T22:51:06.877 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-11T22:51:07.009 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephadm-2:17.2.7-620.gc186d0ca.el9.noarch 1/1 2024-03-11T22:51:07.009 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:07.009 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-11T22:51:07.009 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm-2:17.2.7-620.gc186d0ca.el9.noarch 2024-03-11T22:51:07.009 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:07.009 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-11T22:51:07.011 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-11T22:51:07.076 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-620.gc186d0ca.e 1/1 2024-03-11T22:51:07.100 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-620.gc186d0ca.e 1/1 2024-03-11T22:51:07.101 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:51:07.101 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-11T22:51:07.101 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:07.233 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-immutable-object-cache 2024-03-11T22:51:07.812 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:57 ago on Mon 11 Mar 2024 10:50:10 PM UTC. 2024-03-11T22:51:07.965 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-11T22:51:07.966 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-11T22:51:07.966 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-11T22:51:07.966 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-11T22:51:07.966 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-11T22:51:07.966 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 137 k 2024-03-11T22:51:07.966 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:07.966 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-11T22:51:07.966 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-11T22:51:07.966 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-11T22:51:07.966 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:07.966 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 137 k 2024-03-11T22:51:07.966 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 379 k 2024-03-11T22:51:07.966 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-11T22:51:08.165 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-620.gc186d0ca.e 1/1 2024-03-11T22:51:08.165 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:08.166 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-11T22:51:08.166 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-immutable-object-cache-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:51:08.166 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:08.166 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-11T22:51:08.416 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr 2024-03-11T22:51:08.650 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-immutable-object-cache-17.2.7-620.gc186d0c 200 kB/s | 137 kB 00:00 2024-03-11T22:51:08.650 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:51:08.650 INFO:teuthology.orchestra.run.smithi153.stdout:Total 200 kB/s | 137 kB 00:00 2024-03-11T22:51:08.651 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-11T22:51:08.658 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-11T22:51:08.659 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-11T22:51:08.712 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-11T22:51:08.712 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-11T22:51:08.851 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-11T22:51:08.909 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-620.gc186d0ca.e 1/1 2024-03-11T22:51:08.934 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-620.gc186d0ca.e 1/1 2024-03-11T22:51:08.934 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:51:08.934 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-11T22:51:08.934 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:08.987 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:00:59 ago on Mon 11 Mar 2024 10:50:09 PM UTC. 2024-03-11T22:51:09.084 INFO:teuthology.orchestra.run.smithi163.stdout:Package ceph-mgr-2:17.2.7-620.gc186d0ca.el9.x86_64 is already installed. 2024-03-11T22:51:09.137 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-11T22:51:09.138 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-03-11T22:51:09.138 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-11T22:51:09.211 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr-dashboard 2024-03-11T22:51:09.798 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:00 ago on Mon 11 Mar 2024 10:50:09 PM UTC. 2024-03-11T22:51:09.949 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-620.gc186d0ca.e 1/1 2024-03-11T22:51:09.949 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:09.949 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-11T22:51:09.949 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:51:09.949 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:09.949 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-11T22:51:09.955 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-11T22:51:09.956 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-11T22:51:09.956 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-03-11T22:51:09.956 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-11T22:51:09.956 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-11T22:51:09.956 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-dashboard noarch 2:17.2.7-620.gc186d0ca.el9 ceph-noarch 1.7 M 2024-03-11T22:51:09.957 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-03-11T22:51:09.957 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-grafana-dashboards noarch 2:17.2.7-620.gc186d0ca.el9 ceph-noarch 23 k 2024-03-11T22:51:09.957 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-prometheus-alerts noarch 2:17.2.7-620.gc186d0ca.el9 ceph-noarch 15 k 2024-03-11T22:51:09.957 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-03-11T22:51:09.957 INFO:teuthology.orchestra.run.smithi163.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-11T22:51:09.957 INFO:teuthology.orchestra.run.smithi163.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-11T22:51:09.957 INFO:teuthology.orchestra.run.smithi163.stdout:Installing weak dependencies: 2024-03-11T22:51:09.957 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-03-11T22:51:09.957 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:09.957 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-11T22:51:09.957 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-11T22:51:09.957 INFO:teuthology.orchestra.run.smithi163.stdout:Install 7 Packages 2024-03-11T22:51:09.957 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:09.958 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 2.0 M 2024-03-11T22:51:09.958 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 8.1 M 2024-03-11T22:51:09.958 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-11T22:51:10.203 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr 2024-03-11T22:51:10.747 INFO:teuthology.orchestra.run.smithi163.stdout:(1/7): ceph-grafana-dashboards-17.2.7-620.gc186 73 kB/s | 23 kB 00:00 2024-03-11T22:51:10.772 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:00 ago on Mon 11 Mar 2024 10:50:10 PM UTC. 2024-03-11T22:51:10.789 INFO:teuthology.orchestra.run.smithi163.stdout:(2/7): ceph-prometheus-alerts-17.2.7-620.gc186d 40 kB/s | 15 kB 00:00 2024-03-11T22:51:10.869 INFO:teuthology.orchestra.run.smithi153.stdout:Package ceph-mgr-2:17.2.7-620.gc186d0ca.el9.x86_64 is already installed. 2024-03-11T22:51:10.922 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-11T22:51:10.922 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-11T22:51:10.922 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-11T22:51:10.987 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-dashboard 2024-03-11T22:51:10.997 INFO:teuthology.orchestra.run.smithi163.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 36 kB/s | 9.0 kB 00:00 2024-03-11T22:51:11.206 INFO:teuthology.orchestra.run.smithi163.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 98 kB/s | 41 kB 00:00 2024-03-11T22:51:11.231 INFO:teuthology.orchestra.run.smithi163.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 132 kB/s | 31 kB 00:00 2024-03-11T22:51:11.545 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:01 ago on Mon 11 Mar 2024 10:50:10 PM UTC. 2024-03-11T22:51:11.640 INFO:teuthology.orchestra.run.smithi163.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 435 kB/s | 188 kB 00:00 2024-03-11T22:51:11.695 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-11T22:51:11.695 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-11T22:51:11.695 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-11T22:51:11.696 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-11T22:51:11.696 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-11T22:51:11.696 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard noarch 2:17.2.7-620.gc186d0ca.el9 ceph-noarch 1.7 M 2024-03-11T22:51:11.696 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-11T22:51:11.696 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards noarch 2:17.2.7-620.gc186d0ca.el9 ceph-noarch 23 k 2024-03-11T22:51:11.696 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts noarch 2:17.2.7-620.gc186d0ca.el9 ceph-noarch 15 k 2024-03-11T22:51:11.696 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-03-11T22:51:11.696 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-11T22:51:11.696 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-11T22:51:11.696 INFO:teuthology.orchestra.run.smithi153.stdout:Installing weak dependencies: 2024-03-11T22:51:11.696 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-03-11T22:51:11.696 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:11.697 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-11T22:51:11.697 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-11T22:51:11.697 INFO:teuthology.orchestra.run.smithi153.stdout:Install 7 Packages 2024-03-11T22:51:11.697 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:11.697 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 2.0 M 2024-03-11T22:51:11.697 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 8.1 M 2024-03-11T22:51:11.697 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-11T22:51:11.945 INFO:teuthology.orchestra.run.smithi153.stdout:(1/7): ceph-grafana-dashboards-17.2.7-620.gc186 133 kB/s | 23 kB 00:00 2024-03-11T22:51:11.970 INFO:teuthology.orchestra.run.smithi153.stdout:(2/7): ceph-prometheus-alerts-17.2.7-620.gc186d 73 kB/s | 15 kB 00:00 2024-03-11T22:51:12.229 INFO:teuthology.orchestra.run.smithi153.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 32 kB/s | 9.0 kB 00:00 2024-03-11T22:51:12.382 INFO:teuthology.orchestra.run.smithi163.stdout:(7/7): ceph-mgr-dashboard-17.2.7-620.gc186d0ca. 889 kB/s | 1.7 MB 00:01 2024-03-11T22:51:12.384 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:51:12.384 INFO:teuthology.orchestra.run.smithi163.stdout:Total 842 kB/s | 2.0 MB 00:02 2024-03-11T22:51:12.384 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-11T22:51:12.395 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-11T22:51:12.395 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-11T22:51:12.454 INFO:teuthology.orchestra.run.smithi153.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 85 kB/s | 41 kB 00:00 2024-03-11T22:51:12.465 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-11T22:51:12.466 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-11T22:51:12.479 INFO:teuthology.orchestra.run.smithi153.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 123 kB/s | 31 kB 00:00 2024-03-11T22:51:12.771 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-11T22:51:12.833 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-03-11T22:51:12.867 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-03-11T22:51:12.922 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-03-11T22:51:12.955 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-03-11T22:51:12.963 INFO:teuthology.orchestra.run.smithi153.stdout:(6/7): ceph-mgr-dashboard-17.2.7-620.gc186d0ca. 1.4 MB/s | 1.7 MB 00:01 2024-03-11T22:51:12.983 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-620.gc186d0ca.el9.no 5/7 2024-03-11T22:51:13.213 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-620.gc186d0ca.el9.n 6/7 2024-03-11T22:51:13.252 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-620.gc186d0ca.el9.noarch 7/7 2024-03-11T22:51:13.605 INFO:teuthology.orchestra.run.smithi153.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 164 kB/s | 188 kB 00:01 2024-03-11T22:51:13.607 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:51:13.607 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.0 MB/s | 2.0 MB 00:01 2024-03-11T22:51:13.608 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-11T22:51:13.617 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-11T22:51:13.617 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-11T22:51:13.691 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-11T22:51:13.691 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-11T22:51:13.793 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-620.gc186d0ca.el9.noarch 7/7 2024-03-11T22:51:13.793 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-620.gc186d0ca.el9.n 1/7 2024-03-11T22:51:13.793 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-620.gc186d0ca.el9.noarch 2/7 2024-03-11T22:51:13.793 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-620.gc186d0ca.el9.no 3/7 2024-03-11T22:51:13.794 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-03-11T22:51:13.794 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-03-11T22:51:13.794 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-03-11T22:51:13.995 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-11T22:51:14.056 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-03-11T22:51:14.090 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-03-11T22:51:14.104 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-03-11T22:51:14.104 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:14.104 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-11T22:51:14.105 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-grafana-dashboards-2:17.2.7-620.gc186d0ca.el9.noarch 2024-03-11T22:51:14.105 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-dashboard-2:17.2.7-620.gc186d0ca.el9.noarch 2024-03-11T22:51:14.105 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-prometheus-alerts-2:17.2.7-620.gc186d0ca.el9.noarch 2024-03-11T22:51:14.105 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-03-11T22:51:14.105 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-03-11T22:51:14.105 INFO:teuthology.orchestra.run.smithi163.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-11T22:51:14.105 INFO:teuthology.orchestra.run.smithi163.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-11T22:51:14.105 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:14.105 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-11T22:51:14.145 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-03-11T22:51:14.185 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-03-11T22:51:14.214 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-620.gc186d0ca.el9.no 5/7 2024-03-11T22:51:14.327 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-11T22:51:14.444 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-620.gc186d0ca.el9.n 6/7 2024-03-11T22:51:14.475 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-620.gc186d0ca.el9.noarch 7/7 2024-03-11T22:51:14.887 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:05 ago on Mon 11 Mar 2024 10:50:09 PM UTC. 2024-03-11T22:51:15.005 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-620.gc186d0ca.el9.noarch 7/7 2024-03-11T22:51:15.005 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-620.gc186d0ca.el9.n 1/7 2024-03-11T22:51:15.005 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-620.gc186d0ca.el9.noarch 2/7 2024-03-11T22:51:15.006 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-620.gc186d0ca.el9.no 3/7 2024-03-11T22:51:15.006 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-03-11T22:51:15.006 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-03-11T22:51:15.006 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-03-11T22:51:15.036 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-11T22:51:15.037 INFO:teuthology.orchestra.run.smithi163.stdout:=========================================================================================== 2024-03-11T22:51:15.037 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-03-11T22:51:15.037 INFO:teuthology.orchestra.run.smithi163.stdout:=========================================================================================== 2024-03-11T22:51:15.037 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-11T22:51:15.037 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-620.gc186d0ca.el9 ceph-noarch 7.4 M 2024-03-11T22:51:15.037 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-03-11T22:51:15.037 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-11T22:51:15.037 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-11T22:51:15.038 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-11T22:51:15.038 INFO:teuthology.orchestra.run.smithi163.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-11T22:51:15.038 INFO:teuthology.orchestra.run.smithi163.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-11T22:51:15.038 INFO:teuthology.orchestra.run.smithi163.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-11T22:51:15.038 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-11T22:51:15.038 INFO:teuthology.orchestra.run.smithi163.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-11T22:51:15.038 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-11T22:51:15.038 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-11T22:51:15.038 INFO:teuthology.orchestra.run.smithi163.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-11T22:51:15.038 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:15.038 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-11T22:51:15.038 INFO:teuthology.orchestra.run.smithi163.stdout:=========================================================================================== 2024-03-11T22:51:15.038 INFO:teuthology.orchestra.run.smithi163.stdout:Install 12 Packages 2024-03-11T22:51:15.039 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:15.039 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 37 M 2024-03-11T22:51:15.039 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 209 M 2024-03-11T22:51:15.039 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-11T22:51:15.314 INFO:teuthology.orchestra.run.smithi163.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 190 kB/s | 32 kB 00:00 2024-03-11T22:51:15.315 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-03-11T22:51:15.315 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:15.315 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-11T22:51:15.315 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards-2:17.2.7-620.gc186d0ca.el9.noarch 2024-03-11T22:51:15.315 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard-2:17.2.7-620.gc186d0ca.el9.noarch 2024-03-11T22:51:15.315 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts-2:17.2.7-620.gc186d0ca.el9.noarch 2024-03-11T22:51:15.315 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-03-11T22:51:15.315 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-03-11T22:51:15.315 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-11T22:51:15.316 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-11T22:51:15.316 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:15.316 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-11T22:51:15.356 INFO:teuthology.orchestra.run.smithi163.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 423 kB/s | 17 kB 00:00 2024-03-11T22:51:15.398 INFO:teuthology.orchestra.run.smithi163.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 822 kB/s | 34 kB 00:00 2024-03-11T22:51:15.540 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-11T22:51:15.557 INFO:teuthology.orchestra.run.smithi163.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.3 MB/s | 3.0 MB 00:00 2024-03-11T22:51:15.624 INFO:teuthology.orchestra.run.smithi163.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2024-03-11T22:51:15.724 INFO:teuthology.orchestra.run.smithi163.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 4.8 MB 00:00 2024-03-11T22:51:15.749 INFO:teuthology.orchestra.run.smithi163.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.7 MB/s | 217 kB 00:00 2024-03-11T22:51:15.866 INFO:teuthology.orchestra.run.smithi163.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 6.8 MB/s | 803 kB 00:00 2024-03-11T22:51:16.059 INFO:teuthology.orchestra.run.smithi163.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-03-11T22:51:16.084 INFO:teuthology.orchestra.run.smithi163.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 910 kB/s | 195 kB 00:00 2024-03-11T22:51:16.096 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:06 ago on Mon 11 Mar 2024 10:50:10 PM UTC. 2024-03-11T22:51:16.249 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-11T22:51:16.249 INFO:teuthology.orchestra.run.smithi153.stdout:=========================================================================================== 2024-03-11T22:51:16.250 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-11T22:51:16.250 INFO:teuthology.orchestra.run.smithi153.stdout:=========================================================================================== 2024-03-11T22:51:16.250 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-11T22:51:16.250 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-620.gc186d0ca.el9 ceph-noarch 7.4 M 2024-03-11T22:51:16.250 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-11T22:51:16.250 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-11T22:51:16.250 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-11T22:51:16.251 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-11T22:51:16.251 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-11T22:51:16.251 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-11T22:51:16.251 INFO:teuthology.orchestra.run.smithi153.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-11T22:51:16.251 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-11T22:51:16.251 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-11T22:51:16.251 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-11T22:51:16.251 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-11T22:51:16.251 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-11T22:51:16.251 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:16.251 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-11T22:51:16.251 INFO:teuthology.orchestra.run.smithi153.stdout:=========================================================================================== 2024-03-11T22:51:16.251 INFO:teuthology.orchestra.run.smithi153.stdout:Install 12 Packages 2024-03-11T22:51:16.251 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:16.252 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 37 M 2024-03-11T22:51:16.252 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 209 M 2024-03-11T22:51:16.253 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-11T22:51:16.284 INFO:teuthology.orchestra.run.smithi163.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-03-11T22:51:16.762 INFO:teuthology.orchestra.run.smithi153.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 214 kB/s | 32 kB 00:00 2024-03-11T22:51:16.803 INFO:teuthology.orchestra.run.smithi153.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 418 kB/s | 17 kB 00:00 2024-03-11T22:51:16.845 INFO:teuthology.orchestra.run.smithi153.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 834 kB/s | 34 kB 00:00 2024-03-11T22:51:16.912 INFO:teuthology.orchestra.run.smithi153.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2024-03-11T22:51:16.976 INFO:teuthology.orchestra.run.smithi163.stdout:(12/12): ceph-mgr-diskprediction-local-17.2.7-6 4.0 MB/s | 7.4 MB 00:01 2024-03-11T22:51:16.979 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:51:16.979 INFO:teuthology.orchestra.run.smithi163.stdout:Total 19 MB/s | 37 MB 00:01 2024-03-11T22:51:17.022 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-11T22:51:17.047 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-11T22:51:17.047 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-11T22:51:17.113 INFO:teuthology.orchestra.run.smithi153.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2024-03-11T22:51:17.188 INFO:teuthology.orchestra.run.smithi153.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 4.8 MB 00:00 2024-03-11T22:51:17.213 INFO:teuthology.orchestra.run.smithi153.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-03-11T22:51:17.451 INFO:teuthology.orchestra.run.smithi153.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 60 MB/s | 16 MB 00:00 2024-03-11T22:51:17.477 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-11T22:51:17.477 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-11T22:51:17.705 INFO:teuthology.orchestra.run.smithi153.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 769 kB/s | 195 kB 00:00 2024-03-11T22:51:17.730 INFO:teuthology.orchestra.run.smithi153.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 803 kB 00:00 2024-03-11T22:51:17.806 INFO:teuthology.orchestra.run.smithi153.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 2.4 MB/s | 245 kB 00:00 2024-03-11T22:51:18.181 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-11T22:51:18.257 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-11T22:51:18.280 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-11T22:51:18.374 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-11T22:51:18.417 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-11T22:51:18.466 INFO:teuthology.orchestra.run.smithi153.stdout:(12/12): ceph-mgr-diskprediction-local-17.2.7-6 4.0 MB/s | 7.4 MB 00:01 2024-03-11T22:51:18.466 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:51:18.466 INFO:teuthology.orchestra.run.smithi153.stdout:Total 17 MB/s | 37 MB 00:02 2024-03-11T22:51:18.506 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-11T22:51:18.531 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-11T22:51:18.531 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-11T22:51:18.597 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-11T22:51:18.624 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-11T22:51:18.706 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-11T22:51:18.974 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-11T22:51:18.974 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-11T22:51:19.374 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-11T22:51:19.512 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-11T22:51:19.676 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-11T22:51:19.754 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-11T22:51:19.794 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-11T22:51:19.895 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-11T22:51:19.930 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-11T22:51:20.108 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-11T22:51:20.137 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-11T22:51:20.220 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-11T22:51:20.769 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-11T22:51:20.885 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-11T22:51:21.000 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-11T22:51:21.793 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-11T22:51:21.826 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-620.gc186d0 12/12 2024-03-11T22:51:22.233 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-11T22:51:23.267 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-11T22:51:23.294 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-620.gc186d0 12/12 2024-03-11T22:51:24.391 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-620.gc186d0 12/12 2024-03-11T22:51:24.391 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-620.gc186d0 1/12 2024-03-11T22:51:24.391 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-11T22:51:24.391 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-11T22:51:24.391 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-11T22:51:24.391 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-11T22:51:24.391 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-11T22:51:24.392 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-11T22:51:24.392 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-11T22:51:24.392 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-11T22:51:24.392 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-11T22:51:24.392 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-11T22:51:24.873 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-11T22:51:24.873 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:24.873 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-11T22:51:24.873 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-diskprediction-local-2:17.2.7-620.gc186d0ca.el9.noarch 2024-03-11T22:51:24.873 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-11T22:51:24.874 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-11T22:51:24.874 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-11T22:51:24.874 INFO:teuthology.orchestra.run.smithi163.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-11T22:51:24.874 INFO:teuthology.orchestra.run.smithi163.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-11T22:51:24.874 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-11T22:51:24.874 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-11T22:51:24.874 INFO:teuthology.orchestra.run.smithi163.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-11T22:51:24.874 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-11T22:51:24.875 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-11T22:51:24.875 INFO:teuthology.orchestra.run.smithi163.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-11T22:51:24.875 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:24.875 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-11T22:51:25.235 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr-rook 2024-03-11T22:51:25.722 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-620.gc186d0 12/12 2024-03-11T22:51:25.722 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-620.gc186d0 1/12 2024-03-11T22:51:25.722 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-11T22:51:25.722 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-11T22:51:25.722 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-11T22:51:25.722 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-11T22:51:25.722 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-11T22:51:25.722 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-11T22:51:25.722 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-11T22:51:25.722 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-11T22:51:25.723 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-11T22:51:25.723 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-11T22:51:25.806 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:16 ago on Mon 11 Mar 2024 10:50:09 PM UTC. 2024-03-11T22:51:25.958 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-11T22:51:25.959 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================= 2024-03-11T22:51:25.959 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-03-11T22:51:25.959 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================= 2024-03-11T22:51:25.959 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-11T22:51:25.959 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-rook noarch 2:17.2.7-620.gc186d0ca.el9 ceph-noarch 47 k 2024-03-11T22:51:25.959 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-03-11T22:51:25.959 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-11T22:51:25.960 INFO:teuthology.orchestra.run.smithi163.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-11T22:51:25.960 INFO:teuthology.orchestra.run.smithi163.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-11T22:51:25.960 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-11T22:51:25.960 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-11T22:51:25.960 INFO:teuthology.orchestra.run.smithi163.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-11T22:51:25.960 INFO:teuthology.orchestra.run.smithi163.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-11T22:51:25.960 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-11T22:51:25.960 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-11T22:51:25.960 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-11T22:51:25.960 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-11T22:51:25.960 INFO:teuthology.orchestra.run.smithi163.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-11T22:51:25.960 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:25.960 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-11T22:51:25.960 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================= 2024-03-11T22:51:25.960 INFO:teuthology.orchestra.run.smithi163.stdout:Install 13 Packages 2024-03-11T22:51:25.960 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:25.961 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 2.0 M 2024-03-11T22:51:25.961 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 26 M 2024-03-11T22:51:25.961 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-11T22:51:26.149 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-11T22:51:26.149 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:26.149 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-11T22:51:26.149 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local-2:17.2.7-620.gc186d0ca.el9.noarch 2024-03-11T22:51:26.149 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-11T22:51:26.149 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-11T22:51:26.149 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-11T22:51:26.149 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-11T22:51:26.149 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-11T22:51:26.149 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-11T22:51:26.149 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-11T22:51:26.149 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-11T22:51:26.150 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-11T22:51:26.150 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-11T22:51:26.150 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-11T22:51:26.150 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:26.150 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-11T22:51:26.419 INFO:teuthology.orchestra.run.smithi163.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 170 kB/s | 18 kB 00:00 2024-03-11T22:51:26.461 INFO:teuthology.orchestra.run.smithi163.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 165 kB/s | 24 kB 00:00 2024-03-11T22:51:26.528 INFO:teuthology.orchestra.run.smithi163.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 174 kB 00:00 2024-03-11T22:51:26.545 INFO:teuthology.orchestra.run.smithi163.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-11T22:51:26.574 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-rook 2024-03-11T22:51:26.578 INFO:teuthology.orchestra.run.smithi163.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.2 MB/s | 211 kB 00:00 2024-03-11T22:51:26.604 INFO:teuthology.orchestra.run.smithi163.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 730 kB/s | 43 kB 00:00 2024-03-11T22:51:26.871 INFO:teuthology.orchestra.run.smithi163.stdout:(7/13): ceph-mgr-rook-17.2.7-620.gc186d0ca.el9. 85 kB/s | 47 kB 00:00 2024-03-11T22:51:26.971 INFO:teuthology.orchestra.run.smithi163.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 41 kB/s | 15 kB 00:00 2024-03-11T22:51:27.021 INFO:teuthology.orchestra.run.smithi163.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 73 kB/s | 32 kB 00:00 2024-03-11T22:51:27.151 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:17 ago on Mon 11 Mar 2024 10:50:10 PM UTC. 2024-03-11T22:51:27.213 INFO:teuthology.orchestra.run.smithi163.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 308 kB/s | 59 kB 00:00 2024-03-11T22:51:27.305 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-11T22:51:27.306 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================= 2024-03-11T22:51:27.307 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-11T22:51:27.307 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================= 2024-03-11T22:51:27.307 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-11T22:51:27.307 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook noarch 2:17.2.7-620.gc186d0ca.el9 ceph-noarch 47 k 2024-03-11T22:51:27.307 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-11T22:51:27.307 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-11T22:51:27.307 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-11T22:51:27.307 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-11T22:51:27.307 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-11T22:51:27.307 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-11T22:51:27.307 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-11T22:51:27.307 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-11T22:51:27.308 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-11T22:51:27.308 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-11T22:51:27.308 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-11T22:51:27.308 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-11T22:51:27.308 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-11T22:51:27.308 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:27.308 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-11T22:51:27.308 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================= 2024-03-11T22:51:27.308 INFO:teuthology.orchestra.run.smithi153.stdout:Install 13 Packages 2024-03-11T22:51:27.308 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:27.309 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 2.0 M 2024-03-11T22:51:27.309 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 26 M 2024-03-11T22:51:27.309 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-11T22:51:27.372 INFO:teuthology.orchestra.run.smithi163.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 566 kB/s | 90 kB 00:00 2024-03-11T22:51:27.447 INFO:teuthology.orchestra.run.smithi163.stdout:(12/13): python3-google-auth-2.28.0-1.el9.noarc 352 kB/s | 203 kB 00:00 2024-03-11T22:51:27.514 INFO:teuthology.orchestra.run.smithi163.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.0 MB/s | 1.0 MB 00:00 2024-03-11T22:51:27.517 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:51:27.518 INFO:teuthology.orchestra.run.smithi163.stdout:Total 1.3 MB/s | 2.0 MB 00:01 2024-03-11T22:51:27.518 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-11T22:51:27.530 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-11T22:51:27.531 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-11T22:51:27.560 INFO:teuthology.orchestra.run.smithi153.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 160 kB/s | 18 kB 00:00 2024-03-11T22:51:27.611 INFO:teuthology.orchestra.run.smithi153.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 149 kB/s | 24 kB 00:00 2024-03-11T22:51:27.631 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-11T22:51:27.633 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-11T22:51:27.669 INFO:teuthology.orchestra.run.smithi153.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 174 kB 00:00 2024-03-11T22:51:27.694 INFO:teuthology.orchestra.run.smithi153.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-11T22:51:27.720 INFO:teuthology.orchestra.run.smithi153.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.2 MB/s | 211 kB 00:00 2024-03-11T22:51:27.745 INFO:teuthology.orchestra.run.smithi153.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 858 kB/s | 43 kB 00:00 2024-03-11T22:51:28.012 INFO:teuthology.orchestra.run.smithi153.stdout:(7/13): ceph-mgr-rook-17.2.7-620.gc186d0ca.el9. 84 kB/s | 47 kB 00:00 2024-03-11T22:51:28.037 INFO:teuthology.orchestra.run.smithi153.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 51 kB/s | 15 kB 00:00 2024-03-11T22:51:28.129 INFO:teuthology.orchestra.run.smithi153.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 79 kB/s | 32 kB 00:00 2024-03-11T22:51:28.158 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-11T22:51:28.338 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-11T22:51:28.407 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-11T22:51:28.463 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-11T22:51:28.508 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-11T22:51:28.513 INFO:teuthology.orchestra.run.smithi153.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 154 kB/s | 59 kB 00:00 2024-03-11T22:51:28.549 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-11T22:51:28.647 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-11T22:51:28.753 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 7/13 2024-03-11T22:51:28.810 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-11T22:51:28.988 INFO:teuthology.orchestra.run.smithi153.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 189 kB/s | 90 kB 00:00 2024-03-11T22:51:29.213 INFO:teuthology.orchestra.run.smithi153.stdout:(12/13): python3-google-auth-2.28.0-1.el9.noarc 169 kB/s | 203 kB 00:01 2024-03-11T22:51:29.536 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-11T22:51:29.582 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-11T22:51:29.616 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-11T22:51:29.662 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-11T22:51:29.686 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-rook-2:17.2.7-620.gc186d0ca.el9.noarch 13/13 2024-03-11T22:51:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-620.gc186d0ca.el9.noarch 13/13 2024-03-11T22:51:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-rook-2:17.2.7-620.gc186d0ca.el9.noarch 1/13 2024-03-11T22:51:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-11T22:51:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-11T22:51:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-11T22:51:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-11T22:51:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-11T22:51:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-11T22:51:30.455 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-11T22:51:30.456 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-11T22:51:30.456 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 10/13 2024-03-11T22:51:30.456 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-11T22:51:30.456 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-11T22:51:30.826 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-11T22:51:30.827 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:30.827 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-11T22:51:30.827 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-rook-2:17.2.7-620.gc186d0ca.el9.noarch 2024-03-11T22:51:30.827 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-11T22:51:30.827 INFO:teuthology.orchestra.run.smithi163.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-11T22:51:30.827 INFO:teuthology.orchestra.run.smithi163.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-11T22:51:30.827 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-11T22:51:30.827 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-11T22:51:30.827 INFO:teuthology.orchestra.run.smithi163.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-11T22:51:30.827 INFO:teuthology.orchestra.run.smithi163.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-11T22:51:30.827 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-11T22:51:30.827 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-11T22:51:30.827 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-11T22:51:30.828 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-11T22:51:30.828 INFO:teuthology.orchestra.run.smithi163.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-11T22:51:30.828 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:30.828 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-11T22:51:31.061 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr-cephadm 2024-03-11T22:51:31.625 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:22 ago on Mon 11 Mar 2024 10:50:09 PM UTC. 2024-03-11T22:51:31.776 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-11T22:51:31.776 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================= 2024-03-11T22:51:31.777 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-03-11T22:51:31.777 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================= 2024-03-11T22:51:31.777 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-11T22:51:31.777 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-cephadm noarch 2:17.2.7-620.gc186d0ca.el9 ceph-noarch 122 k 2024-03-11T22:51:31.777 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-03-11T22:51:31.777 INFO:teuthology.orchestra.run.smithi163.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-11T22:51:31.777 INFO:teuthology.orchestra.run.smithi163.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-11T22:51:31.777 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-11T22:51:31.777 INFO:teuthology.orchestra.run.smithi163.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-11T22:51:31.777 INFO:teuthology.orchestra.run.smithi163.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-11T22:51:31.777 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:31.777 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-11T22:51:31.777 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================= 2024-03-11T22:51:31.778 INFO:teuthology.orchestra.run.smithi163.stdout:Install 6 Packages 2024-03-11T22:51:31.778 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:31.778 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 6.8 M 2024-03-11T22:51:31.778 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 33 M 2024-03-11T22:51:31.778 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-11T22:51:31.988 INFO:teuthology.orchestra.run.smithi153.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 265 kB/s | 1.0 MB 00:03 2024-03-11T22:51:31.990 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:51:31.990 INFO:teuthology.orchestra.run.smithi153.stdout:Total 447 kB/s | 2.0 MB 00:04 2024-03-11T22:51:31.991 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-11T22:51:32.005 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-11T22:51:32.006 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-11T22:51:32.112 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-11T22:51:32.113 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-11T22:51:32.643 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-11T22:51:32.784 INFO:teuthology.orchestra.run.smithi163.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-03-11T22:51:32.809 INFO:teuthology.orchestra.run.smithi163.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 765 kB/s | 248 kB 00:00 2024-03-11T22:51:32.819 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-11T22:51:32.872 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-11T22:51:32.935 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-11T22:51:32.965 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-11T22:51:32.968 INFO:teuthology.orchestra.run.smithi163.stdout:(3/6): ceph-mgr-cephadm-17.2.7-620.gc186d0ca.el 252 kB/s | 122 kB 00:00 2024-03-11T22:51:33.006 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-11T22:51:33.099 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-11T22:51:33.185 INFO:teuthology.orchestra.run.smithi163.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 154 kB/s | 58 kB 00:00 2024-03-11T22:51:33.200 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 7/13 2024-03-11T22:51:33.249 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-11T22:51:33.335 INFO:teuthology.orchestra.run.smithi163.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 139 kB/s | 51 kB 00:00 2024-03-11T22:51:33.401 INFO:teuthology.orchestra.run.smithi163.stdout:(6/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 887 kB/s | 548 kB 00:00 2024-03-11T22:51:33.404 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:51:33.405 INFO:teuthology.orchestra.run.smithi163.stdout:Total 4.2 MB/s | 6.8 MB 00:01 2024-03-11T22:51:33.417 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-11T22:51:33.427 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-11T22:51:33.427 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-11T22:51:33.520 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-11T22:51:33.522 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-11T22:51:33.824 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-11T22:51:33.974 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-11T22:51:33.990 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-11T22:51:34.021 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-11T22:51:34.036 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-03-11T22:51:34.055 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-11T22:51:34.100 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-11T22:51:34.125 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-rook-2:17.2.7-620.gc186d0ca.el9.noarch 13/13 2024-03-11T22:51:34.551 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-11T22:51:34.639 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-11T22:51:34.694 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-11T22:51:34.725 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-620.gc186d0ca.el9.noarch 6/6 2024-03-11T22:51:34.948 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-620.gc186d0ca.el9.noarch 13/13 2024-03-11T22:51:34.948 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-rook-2:17.2.7-620.gc186d0ca.el9.noarch 1/13 2024-03-11T22:51:34.948 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-11T22:51:34.948 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-11T22:51:34.948 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-11T22:51:34.948 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-11T22:51:34.948 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-11T22:51:34.948 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-11T22:51:34.948 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-11T22:51:34.948 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-11T22:51:34.948 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 10/13 2024-03-11T22:51:34.949 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-11T22:51:34.949 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-11T22:51:35.303 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-11T22:51:35.304 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:35.304 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-11T22:51:35.304 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook-2:17.2.7-620.gc186d0ca.el9.noarch 2024-03-11T22:51:35.304 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-11T22:51:35.304 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-11T22:51:35.304 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-11T22:51:35.304 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-11T22:51:35.304 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-11T22:51:35.304 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-11T22:51:35.304 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-11T22:51:35.304 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-11T22:51:35.304 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-11T22:51:35.305 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-11T22:51:35.305 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-11T22:51:35.305 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-11T22:51:35.305 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:35.305 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-11T22:51:35.522 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-620.gc186d0ca.el9.noarch 6/6 2024-03-11T22:51:35.522 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-620.gc186d0ca.el9.noarch 1/6 2024-03-11T22:51:35.522 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-11T22:51:35.522 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-11T22:51:35.522 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-03-11T22:51:35.522 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-11T22:51:35.579 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-cephadm 2024-03-11T22:51:35.834 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-11T22:51:35.834 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:35.834 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-11T22:51:35.834 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-cephadm-2:17.2.7-620.gc186d0ca.el9.noarch 2024-03-11T22:51:35.834 INFO:teuthology.orchestra.run.smithi163.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-11T22:51:35.834 INFO:teuthology.orchestra.run.smithi163.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-11T22:51:35.834 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-11T22:51:35.834 INFO:teuthology.orchestra.run.smithi163.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-11T22:51:35.834 INFO:teuthology.orchestra.run.smithi163.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-11T22:51:35.834 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:35.834 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-11T22:51:36.085 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-fuse 2024-03-11T22:51:36.152 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:26 ago on Mon 11 Mar 2024 10:50:10 PM UTC. 2024-03-11T22:51:36.307 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-11T22:51:36.308 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================= 2024-03-11T22:51:36.308 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-11T22:51:36.308 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================= 2024-03-11T22:51:36.308 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-11T22:51:36.308 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm noarch 2:17.2.7-620.gc186d0ca.el9 ceph-noarch 122 k 2024-03-11T22:51:36.309 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-11T22:51:36.309 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-11T22:51:36.309 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-11T22:51:36.309 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-11T22:51:36.309 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-11T22:51:36.309 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-11T22:51:36.309 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:36.309 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-11T22:51:36.309 INFO:teuthology.orchestra.run.smithi153.stdout:======================================================================================= 2024-03-11T22:51:36.309 INFO:teuthology.orchestra.run.smithi153.stdout:Install 6 Packages 2024-03-11T22:51:36.309 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:36.310 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 6.8 M 2024-03-11T22:51:36.310 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 33 M 2024-03-11T22:51:36.310 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-11T22:51:36.648 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:27 ago on Mon 11 Mar 2024 10:50:09 PM UTC. 2024-03-11T22:51:36.739 INFO:teuthology.orchestra.run.smithi153.stdout:(1/6): ceph-mgr-cephadm-17.2.7-620.gc186d0ca.el 546 kB/s | 122 kB 00:00 2024-03-11T22:51:36.798 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-11T22:51:36.799 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-11T22:51:36.799 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-03-11T22:51:36.799 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-11T22:51:36.799 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-11T22:51:36.799 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-fuse x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 812 k 2024-03-11T22:51:36.799 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-03-11T22:51:36.799 INFO:teuthology.orchestra.run.smithi163.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-11T22:51:36.799 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:36.799 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-11T22:51:36.800 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-11T22:51:36.800 INFO:teuthology.orchestra.run.smithi163.stdout:Install 2 Packages 2024-03-11T22:51:36.800 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:36.800 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 891 k 2024-03-11T22:51:36.800 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 2.6 M 2024-03-11T22:51:36.800 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-11T22:51:36.831 INFO:teuthology.orchestra.run.smithi153.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-03-11T22:51:36.946 INFO:teuthology.orchestra.run.smithi163.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 540 kB/s | 79 kB 00:00 2024-03-11T22:51:36.956 INFO:teuthology.orchestra.run.smithi153.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 462 kB/s | 58 kB 00:00 2024-03-11T22:51:37.015 INFO:teuthology.orchestra.run.smithi153.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 874 kB/s | 51 kB 00:00 2024-03-11T22:51:37.040 INFO:teuthology.orchestra.run.smithi153.stdout:(5/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 1.8 MB/s | 548 kB 00:00 2024-03-11T22:51:37.088 INFO:teuthology.orchestra.run.smithi163.stdout:(2/2): ceph-fuse-17.2.7-620.gc186d0ca.el9.x86_6 2.8 MB/s | 812 kB 00:00 2024-03-11T22:51:37.089 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:51:37.090 INFO:teuthology.orchestra.run.smithi163.stdout:Total 3.0 MB/s | 891 kB 00:00 2024-03-11T22:51:37.090 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-11T22:51:37.111 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-11T22:51:37.111 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-11T22:51:37.149 INFO:teuthology.orchestra.run.smithi153.stdout:(6/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 392 kB/s | 248 kB 00:00 2024-03-11T22:51:37.150 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:51:37.150 INFO:teuthology.orchestra.run.smithi153.stdout:Total 8.1 MB/s | 6.8 MB 00:00 2024-03-11T22:51:37.162 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-11T22:51:37.173 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-11T22:51:37.173 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-11T22:51:37.186 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-11T22:51:37.186 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-11T22:51:37.268 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-11T22:51:37.269 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-11T22:51:37.375 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-11T22:51:37.509 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-11T22:51:37.569 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-11T22:51:37.571 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-fuse-2:17.2.7-620.gc186d0ca.el9.x86_64 2/2 2024-03-11T22:51:37.741 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-11T22:51:37.792 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-03-11T22:51:38.238 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-fuse-2:17.2.7-620.gc186d0ca.el9.x86_64 2/2 2024-03-11T22:51:38.238 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-fuse-2:17.2.7-620.gc186d0ca.el9.x86_64 1/2 2024-03-11T22:51:38.310 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-11T22:51:38.395 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-11T22:51:38.450 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-11T22:51:38.479 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-620.gc186d0ca.el9.noarch 6/6 2024-03-11T22:51:38.550 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-11T22:51:38.550 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:38.550 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-11T22:51:38.550 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-fuse-2:17.2.7-620.gc186d0ca.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-11T22:51:38.550 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:38.550 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-11T22:51:38.825 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install librados-devel 2024-03-11T22:51:39.290 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-620.gc186d0ca.el9.noarch 6/6 2024-03-11T22:51:39.291 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-620.gc186d0ca.el9.noarch 1/6 2024-03-11T22:51:39.291 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-11T22:51:39.291 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-11T22:51:39.291 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-03-11T22:51:39.291 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-11T22:51:39.400 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:30 ago on Mon 11 Mar 2024 10:50:09 PM UTC. 2024-03-11T22:51:39.560 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-11T22:51:39.561 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-11T22:51:39.561 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-03-11T22:51:39.561 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-11T22:51:39.561 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-11T22:51:39.561 INFO:teuthology.orchestra.run.smithi163.stdout: librados-devel x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 126 k 2024-03-11T22:51:39.561 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:39.561 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-11T22:51:39.561 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-11T22:51:39.562 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-11T22:51:39.562 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:39.562 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 126 k 2024-03-11T22:51:39.562 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 448 k 2024-03-11T22:51:39.563 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-11T22:51:39.603 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-11T22:51:39.604 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:39.604 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-11T22:51:39.604 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm-2:17.2.7-620.gc186d0ca.el9.noarch 2024-03-11T22:51:39.604 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-11T22:51:39.604 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-11T22:51:39.604 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-11T22:51:39.604 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-11T22:51:39.604 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-11T22:51:39.604 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:39.604 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-11T22:51:39.794 INFO:teuthology.orchestra.run.smithi163.stdout:librados-devel-17.2.7-620.gc186d0ca.el9.x86_64. 543 kB/s | 126 kB 00:00 2024-03-11T22:51:39.797 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:51:39.798 INFO:teuthology.orchestra.run.smithi163.stdout:Total 539 kB/s | 126 kB 00:00 2024-03-11T22:51:39.798 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-11T22:51:39.802 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-11T22:51:39.802 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-11T22:51:39.864 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-11T22:51:39.865 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-11T22:51:39.867 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-fuse 2024-03-11T22:51:40.010 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-11T22:51:40.103 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librados-devel-2:17.2.7-620.gc186d0ca.el9.x86_64 1/1 2024-03-11T22:51:40.438 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:30 ago on Mon 11 Mar 2024 10:50:10 PM UTC. 2024-03-11T22:51:40.583 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados-devel-2:17.2.7-620.gc186d0ca.el9.x86_64 1/1 2024-03-11T22:51:40.589 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-11T22:51:40.589 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-11T22:51:40.589 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-03-11T22:51:40.590 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-11T22:51:40.590 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-11T22:51:40.590 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 812 k 2024-03-11T22:51:40.590 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-11T22:51:40.590 INFO:teuthology.orchestra.run.smithi153.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-11T22:51:40.590 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:40.590 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-11T22:51:40.590 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-11T22:51:40.590 INFO:teuthology.orchestra.run.smithi153.stdout:Install 2 Packages 2024-03-11T22:51:40.590 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:40.591 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 891 k 2024-03-11T22:51:40.591 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 2.6 M 2024-03-11T22:51:40.591 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-11T22:51:40.734 INFO:teuthology.orchestra.run.smithi153.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 551 kB/s | 79 kB 00:00 2024-03-11T22:51:40.866 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados-devel-2:17.2.7-620.gc186d0ca.el9.x86_64 1/1 2024-03-11T22:51:40.866 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:40.867 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-11T22:51:40.867 INFO:teuthology.orchestra.run.smithi163.stdout: librados-devel-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:51:40.867 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:40.867 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-11T22:51:41.117 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install libcephfs2 2024-03-11T22:51:41.459 INFO:teuthology.orchestra.run.smithi153.stdout:(2/2): ceph-fuse-17.2.7-620.gc186d0ca.el9.x86_6 936 kB/s | 812 kB 00:00 2024-03-11T22:51:41.460 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:51:41.460 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.0 MB/s | 891 kB 00:00 2024-03-11T22:51:41.461 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-11T22:51:41.483 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-11T22:51:41.483 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-11T22:51:41.562 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-11T22:51:41.562 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-11T22:51:41.693 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:32 ago on Mon 11 Mar 2024 10:50:09 PM UTC. 2024-03-11T22:51:41.748 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-11T22:51:41.792 INFO:teuthology.orchestra.run.smithi163.stdout:Package libcephfs2-2:17.2.7-620.gc186d0ca.el9.x86_64 is already installed. 2024-03-11T22:51:41.845 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-11T22:51:41.846 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-03-11T22:51:41.846 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-11T22:51:41.871 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-11T22:51:41.924 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install libcephfs-devel 2024-03-11T22:51:41.941 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-fuse-2:17.2.7-620.gc186d0ca.el9.x86_64 2/2 2024-03-11T22:51:42.492 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:33 ago on Mon 11 Mar 2024 10:50:09 PM UTC. 2024-03-11T22:51:42.640 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-11T22:51:42.640 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-11T22:51:42.640 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-03-11T22:51:42.641 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-11T22:51:42.641 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-11T22:51:42.641 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs-devel x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 25 k 2024-03-11T22:51:42.641 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:42.641 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-11T22:51:42.641 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-11T22:51:42.641 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-11T22:51:42.641 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:42.642 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 25 k 2024-03-11T22:51:42.642 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 110 k 2024-03-11T22:51:42.642 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-11T22:51:42.683 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-fuse-2:17.2.7-620.gc186d0ca.el9.x86_64 2/2 2024-03-11T22:51:42.683 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-fuse-2:17.2.7-620.gc186d0ca.el9.x86_64 1/2 2024-03-11T22:51:42.986 INFO:teuthology.orchestra.run.smithi163.stdout:libcephfs-devel-17.2.7-620.gc186d0ca.el9.x86_64 72 kB/s | 25 kB 00:00 2024-03-11T22:51:42.987 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:51:42.987 INFO:teuthology.orchestra.run.smithi163.stdout:Total 72 kB/s | 25 kB 00:00 2024-03-11T22:51:42.987 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-11T22:51:42.991 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-11T22:51:42.991 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-11T22:51:42.992 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-11T22:51:42.993 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-11T22:51:42.993 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-11T22:51:42.993 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:42.993 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-11T22:51:42.993 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse-2:17.2.7-620.gc186d0ca.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-11T22:51:42.993 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:42.994 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-11T22:51:43.084 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-11T22:51:43.191 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephfs-devel-2:17.2.7-620.gc186d0ca.el9.x86_64 1/1 2024-03-11T22:51:43.292 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librados-devel 2024-03-11T22:51:43.569 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-620.gc186d0ca.el9.x86_64 1/1 2024-03-11T22:51:43.848 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephfs-devel-2:17.2.7-620.gc186d0ca.el9.x86_64 1/1 2024-03-11T22:51:43.848 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:43.848 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-11T22:51:43.848 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs-devel-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:51:43.848 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:43.848 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-11T22:51:43.870 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:33 ago on Mon 11 Mar 2024 10:50:10 PM UTC. 2024-03-11T22:51:44.019 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-11T22:51:44.020 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-11T22:51:44.020 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-11T22:51:44.020 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-11T22:51:44.020 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-11T22:51:44.020 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 126 k 2024-03-11T22:51:44.020 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:44.020 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-11T22:51:44.020 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-11T22:51:44.020 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-11T22:51:44.020 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:44.020 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 126 k 2024-03-11T22:51:44.021 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 448 k 2024-03-11T22:51:44.021 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-11T22:51:44.037 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install librados2 2024-03-11T22:51:44.605 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:35 ago on Mon 11 Mar 2024 10:50:09 PM UTC. 2024-03-11T22:51:44.613 INFO:teuthology.orchestra.run.smithi153.stdout:librados-devel-17.2.7-620.gc186d0ca.el9.x86_64. 213 kB/s | 126 kB 00:00 2024-03-11T22:51:44.613 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:51:44.613 INFO:teuthology.orchestra.run.smithi153.stdout:Total 213 kB/s | 126 kB 00:00 2024-03-11T22:51:44.614 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-11T22:51:44.620 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-11T22:51:44.620 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-11T22:51:44.676 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-11T22:51:44.676 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-11T22:51:44.702 INFO:teuthology.orchestra.run.smithi163.stdout:Package librados2-2:17.2.7-620.gc186d0ca.el9.x86_64 is already installed. 2024-03-11T22:51:44.755 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-11T22:51:44.756 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-03-11T22:51:44.756 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-11T22:51:44.817 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-11T22:51:44.829 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install librbd1 2024-03-11T22:51:44.921 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librados-devel-2:17.2.7-620.gc186d0ca.el9.x86_64 1/1 2024-03-11T22:51:45.358 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados-devel-2:17.2.7-620.gc186d0ca.el9.x86_64 1/1 2024-03-11T22:51:45.388 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:36 ago on Mon 11 Mar 2024 10:50:09 PM UTC. 2024-03-11T22:51:45.486 INFO:teuthology.orchestra.run.smithi163.stdout:Package librbd1-2:17.2.7-620.gc186d0ca.el9.x86_64 is already installed. 2024-03-11T22:51:45.538 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-11T22:51:45.539 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-03-11T22:51:45.539 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-11T22:51:45.613 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install python3-rados 2024-03-11T22:51:45.678 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados-devel-2:17.2.7-620.gc186d0ca.el9.x86_64 1/1 2024-03-11T22:51:45.678 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:45.678 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-11T22:51:45.678 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:51:45.678 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:45.678 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-11T22:51:45.925 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install libcephfs2 2024-03-11T22:51:46.180 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:37 ago on Mon 11 Mar 2024 10:50:09 PM UTC. 2024-03-11T22:51:46.277 INFO:teuthology.orchestra.run.smithi163.stdout:Package python3-rados-2:17.2.7-620.gc186d0ca.el9.x86_64 is already installed. 2024-03-11T22:51:46.330 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-11T22:51:46.331 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-03-11T22:51:46.331 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-11T22:51:46.401 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install python3-rgw 2024-03-11T22:51:46.510 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:36 ago on Mon 11 Mar 2024 10:50:10 PM UTC. 2024-03-11T22:51:46.608 INFO:teuthology.orchestra.run.smithi153.stdout:Package libcephfs2-2:17.2.7-620.gc186d0ca.el9.x86_64 is already installed. 2024-03-11T22:51:46.661 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-11T22:51:46.662 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-11T22:51:46.662 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-11T22:51:46.749 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install libcephfs-devel 2024-03-11T22:51:46.968 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:37 ago on Mon 11 Mar 2024 10:50:09 PM UTC. 2024-03-11T22:51:47.066 INFO:teuthology.orchestra.run.smithi163.stdout:Package python3-rgw-2:17.2.7-620.gc186d0ca.el9.x86_64 is already installed. 2024-03-11T22:51:47.119 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-11T22:51:47.120 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-03-11T22:51:47.120 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-11T22:51:47.194 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install python3-cephfs 2024-03-11T22:51:47.335 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:37 ago on Mon 11 Mar 2024 10:50:10 PM UTC. 2024-03-11T22:51:47.487 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-11T22:51:47.487 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-11T22:51:47.487 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-11T22:51:47.487 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-11T22:51:47.487 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-11T22:51:47.487 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 25 k 2024-03-11T22:51:47.487 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:47.488 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-11T22:51:47.488 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-11T22:51:47.488 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-11T22:51:47.488 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:47.488 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 25 k 2024-03-11T22:51:47.488 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 110 k 2024-03-11T22:51:47.488 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-11T22:51:47.760 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:38 ago on Mon 11 Mar 2024 10:50:09 PM UTC. 2024-03-11T22:51:47.835 INFO:teuthology.orchestra.run.smithi153.stdout:libcephfs-devel-17.2.7-620.gc186d0ca.el9.x86_64 72 kB/s | 25 kB 00:00 2024-03-11T22:51:47.835 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:51:47.836 INFO:teuthology.orchestra.run.smithi153.stdout:Total 72 kB/s | 25 kB 00:00 2024-03-11T22:51:47.836 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-11T22:51:47.839 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-11T22:51:47.839 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-11T22:51:47.841 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-11T22:51:47.841 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-11T22:51:47.857 INFO:teuthology.orchestra.run.smithi163.stdout:Package python3-cephfs-2:17.2.7-620.gc186d0ca.el9.x86_64 is already installed. 2024-03-11T22:51:47.910 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-11T22:51:47.912 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-03-11T22:51:47.912 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-11T22:51:47.930 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-11T22:51:47.988 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install python3-rbd 2024-03-11T22:51:48.042 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs-devel-2:17.2.7-620.gc186d0ca.el9.x86_64 1/1 2024-03-11T22:51:48.421 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-620.gc186d0ca.el9.x86_64 1/1 2024-03-11T22:51:48.553 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:39 ago on Mon 11 Mar 2024 10:50:09 PM UTC. 2024-03-11T22:51:48.651 INFO:teuthology.orchestra.run.smithi163.stdout:Package python3-rbd-2:17.2.7-620.gc186d0ca.el9.x86_64 is already installed. 2024-03-11T22:51:48.674 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs-devel-2:17.2.7-620.gc186d0ca.el9.x86_64 1/1 2024-03-11T22:51:48.675 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:48.675 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-11T22:51:48.675 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:51:48.675 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:48.675 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-11T22:51:48.704 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-11T22:51:48.706 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-03-11T22:51:48.706 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-11T22:51:48.782 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install rbd-fuse 2024-03-11T22:51:48.849 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librados2 2024-03-11T22:51:49.345 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:40 ago on Mon 11 Mar 2024 10:50:09 PM UTC. 2024-03-11T22:51:49.408 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:39 ago on Mon 11 Mar 2024 10:50:10 PM UTC. 2024-03-11T22:51:49.495 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-11T22:51:49.495 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-11T22:51:49.495 INFO:teuthology.orchestra.run.smithi163.stdout: Package Architecture Version Repository Size 2024-03-11T22:51:49.495 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-11T22:51:49.496 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-11T22:51:49.496 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-fuse x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 81 k 2024-03-11T22:51:49.496 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:49.496 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-11T22:51:49.496 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-11T22:51:49.496 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-11T22:51:49.496 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:49.496 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 81 k 2024-03-11T22:51:49.496 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 222 k 2024-03-11T22:51:49.496 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-11T22:51:49.506 INFO:teuthology.orchestra.run.smithi153.stdout:Package librados2-2:17.2.7-620.gc186d0ca.el9.x86_64 is already installed. 2024-03-11T22:51:49.559 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-11T22:51:49.560 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-11T22:51:49.560 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-11T22:51:49.628 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librbd1 2024-03-11T22:51:50.004 INFO:teuthology.orchestra.run.smithi163.stdout:rbd-fuse-17.2.7-620.gc186d0ca.el9.x86_64.rpm 160 kB/s | 81 kB 00:00 2024-03-11T22:51:50.005 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:51:50.005 INFO:teuthology.orchestra.run.smithi163.stdout:Total 160 kB/s | 81 kB 00:00 2024-03-11T22:51:50.005 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-11T22:51:50.013 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-11T22:51:50.013 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-11T22:51:50.068 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-11T22:51:50.068 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-11T22:51:50.187 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:40 ago on Mon 11 Mar 2024 10:50:10 PM UTC. 2024-03-11T22:51:50.220 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-11T22:51:50.284 INFO:teuthology.orchestra.run.smithi153.stdout:Package librbd1-2:17.2.7-620.gc186d0ca.el9.x86_64 is already installed. 2024-03-11T22:51:50.319 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-fuse-2:17.2.7-620.gc186d0ca.el9.x86_64 1/1 2024-03-11T22:51:50.336 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-11T22:51:50.337 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-11T22:51:50.337 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-11T22:51:50.401 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rados 2024-03-11T22:51:50.770 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: rbd-fuse-2:17.2.7-620.gc186d0ca.el9.x86_64 1/1 2024-03-11T22:51:50.968 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:40 ago on Mon 11 Mar 2024 10:50:10 PM UTC. 2024-03-11T22:51:51.064 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rados-2:17.2.7-620.gc186d0ca.el9.x86_64 is already installed. 2024-03-11T22:51:51.074 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-fuse-2:17.2.7-620.gc186d0ca.el9.x86_64 1/1 2024-03-11T22:51:51.075 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:51.075 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-11T22:51:51.075 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-fuse-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:51:51.075 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:51.075 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-11T22:51:51.116 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-11T22:51:51.117 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-11T22:51:51.117 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-11T22:51:51.186 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rgw 2024-03-11T22:51:51.315 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install rbd-mirror 2024-03-11T22:51:51.750 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:41 ago on Mon 11 Mar 2024 10:50:10 PM UTC. 2024-03-11T22:51:51.848 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rgw-2:17.2.7-620.gc186d0ca.el9.x86_64 is already installed. 2024-03-11T22:51:51.891 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:42 ago on Mon 11 Mar 2024 10:50:09 PM UTC. 2024-03-11T22:51:51.900 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-11T22:51:51.902 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-11T22:51:51.902 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-11T22:51:51.977 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-cephfs 2024-03-11T22:51:52.050 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-11T22:51:52.051 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-11T22:51:52.051 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-03-11T22:51:52.051 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-11T22:51:52.051 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-11T22:51:52.051 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-mirror x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 2.9 M 2024-03-11T22:51:52.051 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:52.051 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-11T22:51:52.051 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-11T22:51:52.051 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-11T22:51:52.051 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:52.051 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 2.9 M 2024-03-11T22:51:52.051 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 12 M 2024-03-11T22:51:52.051 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-11T22:51:52.417 INFO:teuthology.orchestra.run.smithi163.stdout:rbd-mirror-17.2.7-620.gc186d0ca.el9.x86_64.rpm 8.0 MB/s | 2.9 MB 00:00 2024-03-11T22:51:52.418 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:51:52.418 INFO:teuthology.orchestra.run.smithi163.stdout:Total 8.0 MB/s | 2.9 MB 00:00 2024-03-11T22:51:52.418 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-11T22:51:52.427 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-11T22:51:52.427 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-11T22:51:52.498 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-11T22:51:52.498 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-11T22:51:52.544 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:42 ago on Mon 11 Mar 2024 10:50:10 PM UTC. 2024-03-11T22:51:52.641 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-cephfs-2:17.2.7-620.gc186d0ca.el9.x86_64 is already installed. 2024-03-11T22:51:52.694 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-11T22:51:52.695 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-11T22:51:52.695 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-11T22:51:52.772 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rbd 2024-03-11T22:51:52.902 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-11T22:51:52.968 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-mirror-2:17.2.7-620.gc186d0ca.el9.x86_64 1/1 2024-03-11T22:51:52.992 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: rbd-mirror-2:17.2.7-620.gc186d0ca.el9.x86_64 1/1 2024-03-11T22:51:52.992 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:51:52.992 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-11T22:51:52.993 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T22:51:52.993 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T22:51:52.993 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:53.339 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:43 ago on Mon 11 Mar 2024 10:50:10 PM UTC. 2024-03-11T22:51:53.438 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rbd-2:17.2.7-620.gc186d0ca.el9.x86_64 is already installed. 2024-03-11T22:51:53.491 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-11T22:51:53.492 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-03-11T22:51:53.492 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-11T22:51:53.564 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-fuse 2024-03-11T22:51:54.105 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-mirror-2:17.2.7-620.gc186d0ca.el9.x86_64 1/1 2024-03-11T22:51:54.106 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:54.106 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-11T22:51:54.106 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-mirror-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:51:54.106 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:54.106 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-11T22:51:54.133 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:44 ago on Mon 11 Mar 2024 10:50:10 PM UTC. 2024-03-11T22:51:54.284 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-11T22:51:54.284 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-11T22:51:54.284 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-03-11T22:51:54.285 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-11T22:51:54.285 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-11T22:51:54.285 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 81 k 2024-03-11T22:51:54.285 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:54.285 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-11T22:51:54.285 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-11T22:51:54.285 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-11T22:51:54.285 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:54.285 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 81 k 2024-03-11T22:51:54.285 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 222 k 2024-03-11T22:51:54.285 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-11T22:51:54.354 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install rbd-nbd 2024-03-11T22:51:54.932 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:45 ago on Mon 11 Mar 2024 10:50:09 PM UTC. 2024-03-11T22:51:54.946 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-fuse-17.2.7-620.gc186d0ca.el9.x86_64.rpm 123 kB/s | 81 kB 00:00 2024-03-11T22:51:54.947 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:51:54.947 INFO:teuthology.orchestra.run.smithi153.stdout:Total 123 kB/s | 81 kB 00:00 2024-03-11T22:51:54.947 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-11T22:51:54.954 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-11T22:51:54.955 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-11T22:51:55.012 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-11T22:51:55.012 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-11T22:51:55.084 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-11T22:51:55.089 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-11T22:51:55.089 INFO:teuthology.orchestra.run.smithi163.stdout: Package Architecture Version Repository Size 2024-03-11T22:51:55.090 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-11T22:51:55.090 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-11T22:51:55.090 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-nbd x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 164 k 2024-03-11T22:51:55.090 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:55.090 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-11T22:51:55.090 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-11T22:51:55.090 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-11T22:51:55.090 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:55.090 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 164 k 2024-03-11T22:51:55.090 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 477 k 2024-03-11T22:51:55.090 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-11T22:51:55.147 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-11T22:51:55.235 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-fuse-2:17.2.7-620.gc186d0ca.el9.x86_64 1/1 2024-03-11T22:51:55.556 INFO:teuthology.orchestra.run.smithi163.stdout:rbd-nbd-17.2.7-620.gc186d0ca.el9.x86_64.rpm 351 kB/s | 164 kB 00:00 2024-03-11T22:51:55.556 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:51:55.556 INFO:teuthology.orchestra.run.smithi163.stdout:Total 350 kB/s | 164 kB 00:00 2024-03-11T22:51:55.556 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-11T22:51:55.564 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-11T22:51:55.564 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-11T22:51:55.621 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-11T22:51:55.622 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-11T22:51:55.640 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-fuse-2:17.2.7-620.gc186d0ca.el9.x86_64 1/1 2024-03-11T22:51:55.776 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-11T22:51:55.895 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-nbd-2:17.2.7-620.gc186d0ca.el9.x86_64 1/1 2024-03-11T22:51:55.943 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-fuse-2:17.2.7-620.gc186d0ca.el9.x86_64 1/1 2024-03-11T22:51:55.944 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:55.944 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-11T22:51:55.944 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:51:55.944 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:55.944 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-11T22:51:56.201 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-mirror 2024-03-11T22:51:56.335 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: rbd-nbd-2:17.2.7-620.gc186d0ca.el9.x86_64 1/1 2024-03-11T22:51:56.638 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-nbd-2:17.2.7-620.gc186d0ca.el9.x86_64 1/1 2024-03-11T22:51:56.638 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:56.638 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-11T22:51:56.638 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-nbd-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:51:56.638 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:51:56.638 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-11T22:51:56.778 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:46 ago on Mon 11 Mar 2024 10:50:10 PM UTC. 2024-03-11T22:51:56.884 DEBUG:teuthology.parallel:result is None 2024-03-11T22:51:56.932 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-11T22:51:56.933 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-11T22:51:56.933 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-03-11T22:51:56.933 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-11T22:51:56.933 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-11T22:51:56.933 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 2.9 M 2024-03-11T22:51:56.933 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:56.933 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-11T22:51:56.933 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-11T22:51:56.933 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-11T22:51:56.933 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:56.934 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 2.9 M 2024-03-11T22:51:56.934 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 12 M 2024-03-11T22:51:56.934 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-11T22:51:57.949 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-mirror-17.2.7-620.gc186d0ca.el9.x86_64.rpm 2.9 MB/s | 2.9 MB 00:01 2024-03-11T22:51:57.949 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:51:57.950 INFO:teuthology.orchestra.run.smithi153.stdout:Total 2.9 MB/s | 2.9 MB 00:01 2024-03-11T22:51:57.950 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-11T22:51:57.959 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-11T22:51:57.959 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-11T22:51:58.032 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-11T22:51:58.032 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-11T22:51:58.438 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-11T22:51:58.508 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-mirror-2:17.2.7-620.gc186d0ca.el9.x86_64 1/1 2024-03-11T22:51:58.532 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-mirror-2:17.2.7-620.gc186d0ca.el9.x86_64 1/1 2024-03-11T22:51:58.532 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T22:51:58.532 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-11T22:51:58.532 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T22:51:58.533 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T22:51:58.533 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:59.722 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-mirror-2:17.2.7-620.gc186d0ca.el9.x86_64 1/1 2024-03-11T22:51:59.722 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:59.722 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-11T22:51:59.722 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:51:59.723 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:51:59.723 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-11T22:51:59.969 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-nbd 2024-03-11T22:52:00.538 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:50 ago on Mon 11 Mar 2024 10:50:10 PM UTC. 2024-03-11T22:52:00.693 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-11T22:52:00.694 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-11T22:52:00.694 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-03-11T22:52:00.694 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-11T22:52:00.694 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-11T22:52:00.694 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd x86_64 2:17.2.7-620.gc186d0ca.el9 ceph 164 k 2024-03-11T22:52:00.694 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:52:00.694 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-11T22:52:00.694 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-11T22:52:00.695 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-03-11T22:52:00.695 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:52:00.695 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 164 k 2024-03-11T22:52:00.695 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 477 k 2024-03-11T22:52:00.695 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-11T22:52:01.327 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-nbd-17.2.7-620.gc186d0ca.el9.x86_64.rpm 260 kB/s | 164 kB 00:00 2024-03-11T22:52:01.327 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:52:01.328 INFO:teuthology.orchestra.run.smithi153.stdout:Total 260 kB/s | 164 kB 00:00 2024-03-11T22:52:01.328 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-11T22:52:01.335 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-11T22:52:01.335 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-11T22:52:01.392 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-11T22:52:01.392 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-11T22:52:01.572 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-11T22:52:01.674 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-nbd-2:17.2.7-620.gc186d0ca.el9.x86_64 1/1 2024-03-11T22:52:02.109 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-nbd-2:17.2.7-620.gc186d0ca.el9.x86_64 1/1 2024-03-11T22:52:02.389 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-nbd-2:17.2.7-620.gc186d0ca.el9.x86_64 1/1 2024-03-11T22:52:02.389 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:52:02.389 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-11T22:52:02.389 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd-2:17.2.7-620.gc186d0ca.el9.x86_64 2024-03-11T22:52:02.389 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:52:02.389 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-11T22:52:02.657 DEBUG:teuthology.parallel:result is None 2024-03-11T22:52:02.657 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-11T22:52:02.658 INFO:teuthology.packaging:ref: None 2024-03-11T22:52:02.658 INFO:teuthology.packaging:tag: None 2024-03-11T22:52:02.658 INFO:teuthology.packaging:branch: quincy 2024-03-11T22:52:02.658 INFO:teuthology.packaging:sha1: 94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:52:02.658 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-03-11T22:52:02.875 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-11T22:52:02.896 INFO:teuthology.orchestra.run.smithi153.stdout:17.2.7-620.gc186d0ca.el9 2024-03-11T22:52:02.896 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-620.gc186d0ca.el9 2024-03-11T22:52:02.896 INFO:teuthology.task.install:The correct ceph version 17.2.7-620.gc186d0ca is installed. 2024-03-11T22:52:02.898 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-11T22:52:02.898 INFO:teuthology.packaging:ref: None 2024-03-11T22:52:02.898 INFO:teuthology.packaging:tag: None 2024-03-11T22:52:02.898 INFO:teuthology.packaging:branch: quincy 2024-03-11T22:52:02.898 INFO:teuthology.packaging:sha1: 94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:52:02.898 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-03-11T22:52:03.093 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-11T22:52:03.118 INFO:teuthology.orchestra.run.smithi163.stdout:17.2.7-620.gc186d0ca.el9 2024-03-11T22:52:03.118 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-620.gc186d0ca.el9 2024-03-11T22:52:03.118 INFO:teuthology.task.install:The correct ceph version 17.2.7-620.gc186d0ca is installed. 2024-03-11T22:52:03.120 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-11T22:52:03.120 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-11T22:52:03.120 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-11T22:52:03.148 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-11T22:52:03.148 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-11T22:52:03.190 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-11T22:52:03.190 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-11T22:52:03.190 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/daemon-helper 2024-03-11T22:52:03.216 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-11T22:52:03.284 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-11T22:52:03.284 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/daemon-helper 2024-03-11T22:52:03.314 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-11T22:52:03.383 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-11T22:52:03.384 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-11T22:52:03.384 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-11T22:52:03.410 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-11T22:52:03.478 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-11T22:52:03.478 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-11T22:52:03.506 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-11T22:52:03.574 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-11T22:52:03.575 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-11T22:52:03.575 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/stdin-killer 2024-03-11T22:52:03.602 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-11T22:52:03.669 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-11T22:52:03.669 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/stdin-killer 2024-03-11T22:52:03.697 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-11T22:52:03.766 INFO:teuthology.run_tasks:Running task print... 2024-03-11T22:52:03.774 INFO:teuthology.task.print:**** done install task... 2024-03-11T22:52:03.774 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-11T22:52:03.873 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '94a539707733325322964c6a187c6a1b41be1daf'} 2024-03-11T22:52:03.873 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-03-11T22:52:03.873 INFO:tasks.cephadm:Cluster fsid is fa913b96-dff9-11ee-95c8-87774f69a715 2024-03-11T22:52:03.873 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-11T22:52:03.873 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-03-11T22:52:03.874 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi153': '172.21.15.153', 'mon.smithi163': '172.21.15.163'} 2024-03-11T22:52:03.874 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-11T22:52:03.874 DEBUG:teuthology.orchestra.run.smithi153:> sudo hostname $(hostname -s) 2024-03-11T22:52:03.902 DEBUG:teuthology.orchestra.run.smithi163:> sudo hostname $(hostname -s) 2024-03-11T22:52:03.930 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-03-11T22:52:03.931 DEBUG:teuthology.orchestra.run.smithi153:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T22:52:04.107 INFO:teuthology.orchestra.run.smithi153.stdout:-rw-r--r--. 1 ubuntu ubuntu 372315 Mar 11 22:52 /home/ubuntu/cephtest/cephadm 2024-03-11T22:52:04.108 DEBUG:teuthology.orchestra.run.smithi163:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T22:52:04.350 INFO:teuthology.orchestra.run.smithi163.stdout:-rw-r--r--. 1 ubuntu ubuntu 372315 Mar 11 22:52 /home/ubuntu/cephtest/cephadm 2024-03-11T22:52:04.350 DEBUG:teuthology.orchestra.run.smithi153:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T22:52:04.366 DEBUG:teuthology.orchestra.run.smithi163:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T22:52:04.393 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-03-11T22:52:04.393 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-03-11T22:52:04.409 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-03-11T22:52:05.183 INFO:teuthology.orchestra.run.smithi163.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-03-11T22:52:05.312 INFO:teuthology.orchestra.run.smithi153.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-03-11T22:53:02.413 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-03-11T22:53:02.413 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_version": "ceph version 17.2.7-620-gc186d0ca (c186d0ca78be18052defec2e0c064f7869af6d3f) quincy (stable)", 2024-03-11T22:53:02.414 INFO:teuthology.orchestra.run.smithi163.stdout: "image_id": "a92afbb1c2834deb6f1bddda3f3c554061dd3ec88e0495c17c1373d9018e466b", 2024-03-11T22:53:02.414 INFO:teuthology.orchestra.run.smithi163.stdout: "repo_digests": [ 2024-03-11T22:53:02.414 INFO:teuthology.orchestra.run.smithi163.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:ebd35aa0069fa2bbcdd3e20c6fffec19bb7fa44bfa11066e4a72109a40218c32" 2024-03-11T22:53:02.414 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-03-11T22:53:02.414 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-03-11T22:53:05.144 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-03-11T22:53:05.144 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_version": "ceph version 17.2.7-620-gc186d0ca (c186d0ca78be18052defec2e0c064f7869af6d3f) quincy (stable)", 2024-03-11T22:53:05.144 INFO:teuthology.orchestra.run.smithi153.stdout: "image_id": "a92afbb1c2834deb6f1bddda3f3c554061dd3ec88e0495c17c1373d9018e466b", 2024-03-11T22:53:05.145 INFO:teuthology.orchestra.run.smithi153.stdout: "repo_digests": [ 2024-03-11T22:53:05.145 INFO:teuthology.orchestra.run.smithi153.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:ebd35aa0069fa2bbcdd3e20c6fffec19bb7fa44bfa11066e4a72109a40218c32" 2024-03-11T22:53:05.145 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-03-11T22:53:05.145 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-03-11T22:53:05.163 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /etc/ceph 2024-03-11T22:53:05.224 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /etc/ceph 2024-03-11T22:53:05.258 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 777 /etc/ceph 2024-03-11T22:53:05.285 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 777 /etc/ceph 2024-03-11T22:53:05.328 INFO:tasks.cephadm:Writing seed config... 2024-03-11T22:53:05.329 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-03-11T22:53:05.329 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-03-11T22:53:05.329 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-03-11T22:53:05.329 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-03-11T22:53:05.329 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-03-11T22:53:05.329 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-11T22:53:05.329 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-11T22:53:05.329 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-03-11T22:53:05.329 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-03-11T22:53:05.330 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-11T22:53:05.330 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-11T22:53:05.330 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-03-11T22:53:05.330 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-11T22:53:05.330 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-11T22:53:05.330 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-11T22:53:05.330 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-11T22:53:05.330 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-11T22:53:05.330 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-11T22:53:05.330 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-11T22:53:05.330 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-11T22:53:05.330 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-11T22:53:05.330 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-11T22:53:05.330 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-11T22:53:05.331 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-03-11T22:53:05.331 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-11T22:53:05.331 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-11T22:53:05.331 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-11T22:53:05.331 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-11T22:53:05.331 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-11T22:53:05.331 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-11T22:53:05.331 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-11T22:53:05.331 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-11T22:53:05.331 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-11T22:53:05.331 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-11T22:53:05.331 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-11T22:53:05.331 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-11T22:53:05.331 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-11T22:53:05.332 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-11T22:53:05.332 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-11T22:53:05.332 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-11T22:53:05.332 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-11T22:53:05.333 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-11T22:53:05.348 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = fa913b96-dff9-11ee-95c8-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-11T22:53:05.348 DEBUG:teuthology.orchestra.run.smithi153:mon.smithi153> sudo journalctl -f -n 0 -u ceph-fa913b96-dff9-11ee-95c8-87774f69a715@mon.smithi153.service 2024-03-11T22:53:05.390 INFO:tasks.cephadm:Bootstrapping... 2024-03-11T22:53:05.390 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid fa913b96-dff9-11ee-95c8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.153 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:53:05.596 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:53:05.596 INFO:teuthology.orchestra.run.smithi153.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', 'fa913b96-dff9-11ee-95c8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.153', '--skip-admin-label'] 2024-03-11T22:53:05.622 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T22:53:05.623 INFO:teuthology.orchestra.run.smithi153.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-11T22:53:05.623 INFO:teuthology.orchestra.run.smithi153.stdout:Verifying podman|docker is present... 2024-03-11T22:53:05.645 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T22:53:05.645 INFO:teuthology.orchestra.run.smithi153.stdout:Verifying lvm2 is present... 2024-03-11T22:53:05.645 INFO:teuthology.orchestra.run.smithi153.stdout:Verifying time synchronization is in place... 2024-03-11T22:53:05.653 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-11T22:53:05.653 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-11T22:53:05.659 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-11T22:53:05.659 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout inactive 2024-03-11T22:53:05.666 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout enabled 2024-03-11T22:53:05.672 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout active 2024-03-11T22:53:05.672 INFO:teuthology.orchestra.run.smithi153.stdout:Unit chronyd.service is enabled and running 2024-03-11T22:53:05.672 INFO:teuthology.orchestra.run.smithi153.stdout:Repeating the final host check... 2024-03-11T22:53:05.696 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T22:53:05.696 INFO:teuthology.orchestra.run.smithi153.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-11T22:53:05.696 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl is present 2024-03-11T22:53:05.696 INFO:teuthology.orchestra.run.smithi153.stdout:lvcreate is present 2024-03-11T22:53:05.703 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-11T22:53:05.703 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-11T22:53:05.708 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-11T22:53:05.709 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout inactive 2024-03-11T22:53:05.715 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout enabled 2024-03-11T22:53:05.721 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout active 2024-03-11T22:53:05.722 INFO:teuthology.orchestra.run.smithi153.stdout:Unit chronyd.service is enabled and running 2024-03-11T22:53:05.722 INFO:teuthology.orchestra.run.smithi153.stdout:Host looks OK 2024-03-11T22:53:05.722 INFO:teuthology.orchestra.run.smithi153.stdout:Cluster fsid: fa913b96-dff9-11ee-95c8-87774f69a715 2024-03-11T22:53:05.722 INFO:teuthology.orchestra.run.smithi153.stdout:Acquiring lock 139878865502608 on /run/cephadm/fa913b96-dff9-11ee-95c8-87774f69a715.lock 2024-03-11T22:53:05.722 INFO:teuthology.orchestra.run.smithi153.stdout:Lock 139878865502608 acquired on /run/cephadm/fa913b96-dff9-11ee-95c8-87774f69a715.lock 2024-03-11T22:53:05.723 INFO:teuthology.orchestra.run.smithi153.stdout:Verifying IP 172.21.15.153 port 3300 ... 2024-03-11T22:53:05.723 INFO:teuthology.orchestra.run.smithi153.stdout:Verifying IP 172.21.15.153 port 6789 ... 2024-03-11T22:53:05.723 INFO:teuthology.orchestra.run.smithi153.stdout:Base mon IP(s) is [172.21.15.153:3300, 172.21.15.153:6789], mon addrv is [v2:172.21.15.153:3300,v1:172.21.15.153:6789] 2024-03-11T22:53:05.726 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.153 metric 100 2024-03-11T22:53:05.726 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.153 metric 100 2024-03-11T22:53:05.729 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-11T22:53:05.729 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-11T22:53:05.729 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1792sec hoplimit 64 pref medium 2024-03-11T22:53:05.732 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-11T22:53:05.732 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-11T22:53:05.732 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-11T22:53:05.732 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-11T22:53:05.732 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:6deb/64 scope link 2024-03-11T22:53:05.732 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-11T22:53:05.733 INFO:teuthology.orchestra.run.smithi153.stdout:Mon IP `172.21.15.153` is in CIDR network `172.21.0.0/20` 2024-03-11T22:53:05.733 INFO:teuthology.orchestra.run.smithi153.stdout:Mon IP `172.21.15.153` is in CIDR network `172.21.0.0/20` 2024-03-11T22:53:05.733 INFO:teuthology.orchestra.run.smithi153.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-11T22:53:05.733 INFO:teuthology.orchestra.run.smithi153.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-11T22:53:05.734 INFO:teuthology.orchestra.run.smithi153.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-03-11T22:53:06.227 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stdout a92afbb1c2834deb6f1bddda3f3c554061dd3ec88e0495c17c1373d9018e466b 2024-03-11T22:53:06.227 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-03-11T22:53:06.227 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stderr Getting image source signatures 2024-03-11T22:53:06.227 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-11T22:53:06.227 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stderr Copying blob sha256:603c7b4ee672c5e5d2a3390a6713981f39fa8f931a61dfcb5f021ff93da76198 2024-03-11T22:53:06.227 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stderr Copying config sha256:a92afbb1c2834deb6f1bddda3f3c554061dd3ec88e0495c17c1373d9018e466b 2024-03-11T22:53:06.228 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-11T22:53:07.348 INFO:teuthology.orchestra.run.smithi153.stdout:ceph: stdout ceph version 17.2.7-620-gc186d0ca (c186d0ca78be18052defec2e0c064f7869af6d3f) quincy (stable) 2024-03-11T22:53:07.349 INFO:teuthology.orchestra.run.smithi153.stdout:Ceph version: ceph version 17.2.7-620-gc186d0ca (c186d0ca78be18052defec2e0c064f7869af6d3f) quincy (stable) 2024-03-11T22:53:07.349 INFO:teuthology.orchestra.run.smithi153.stdout:Extracting ceph user uid/gid from container image... 2024-03-11T22:53:08.355 INFO:teuthology.orchestra.run.smithi153.stdout:stat: stdout 167 167 2024-03-11T22:53:08.355 INFO:teuthology.orchestra.run.smithi153.stdout:Creating initial keys... 2024-03-11T22:53:09.243 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph-authtool: stdout AQBUi+9lhmIsJxAAdTpveq5LR5g7vgj2ZSkEFg== 2024-03-11T22:53:10.116 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph-authtool: stdout AQBVi+9lmcGHJhAA6S/6BmZ0dtyCnQc5g8HY9A== 2024-03-11T22:53:10.973 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph-authtool: stdout AQBWi+9lMELiGBAAHajAP94lrnYVd7H/hK9uyA== 2024-03-11T22:53:10.974 INFO:teuthology.orchestra.run.smithi153.stdout:Creating initial monmap... 2024-03-11T22:53:11.995 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-11T22:53:11.995 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-03-11T22:53:11.996 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to fa913b96-dff9-11ee-95c8-87774f69a715 2024-03-11T22:53:11.996 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-11T22:53:11.996 INFO:teuthology.orchestra.run.smithi153.stdout:monmaptool for smithi153 [v2:172.21.15.153:3300,v1:172.21.15.153:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-11T22:53:11.996 INFO:teuthology.orchestra.run.smithi153.stdout:setting min_mon_release = octopus 2024-03-11T22:53:11.996 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/monmaptool: set fsid to fa913b96-dff9-11ee-95c8-87774f69a715 2024-03-11T22:53:11.996 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-11T22:53:11.996 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:53:11.996 INFO:teuthology.orchestra.run.smithi153.stdout:Creating mon... 2024-03-11T22:53:12.901 INFO:teuthology.orchestra.run.smithi153.stdout:create mon.smithi153 on 2024-03-11T22:53:13.106 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-11T22:53:13.264 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-11T22:53:13.419 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-fa913b96-dff9-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-fa913b96-dff9-11ee-95c8-87774f69a715.target. 2024-03-11T22:53:13.419 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-fa913b96-dff9-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-fa913b96-dff9-11ee-95c8-87774f69a715.target. 2024-03-11T22:53:13.608 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-fa913b96-dff9-11ee-95c8-87774f69a715@mon.smithi153 2024-03-11T22:53:13.608 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Failed to reset failed state of unit ceph-fa913b96-dff9-11ee-95c8-87774f69a715@mon.smithi153.service: Unit ceph-fa913b96-dff9-11ee-95c8-87774f69a715@mon.smithi153.service not loaded. 2024-03-11T22:53:13.779 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-fa913b96-dff9-11ee-95c8-87774f69a715.target.wants/ceph-fa913b96-dff9-11ee-95c8-87774f69a715@mon.smithi153.service → /etc/systemd/system/ceph-fa913b96-dff9-11ee-95c8-87774f69a715@.service. 2024-03-11T22:53:14.336 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T22:53:14.337 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout disabled 2024-03-11T22:53:14.345 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T22:53:14.345 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout inactive 2024-03-11T22:53:14.345 INFO:teuthology.orchestra.run.smithi153.stdout:firewalld.service is not enabled 2024-03-11T22:53:14.345 INFO:teuthology.orchestra.run.smithi153.stdout:Not possible to enable service . firewalld.service is not available 2024-03-11T22:53:14.346 INFO:teuthology.orchestra.run.smithi153.stdout:Waiting for mon to start... 2024-03-11T22:53:14.346 INFO:teuthology.orchestra.run.smithi153.stdout:Waiting for mon... 2024-03-11T22:53:15.427 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:15 smithi153 ceph-mon[28589]: from='client.? 172.21.15.153:0/2332429226' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-11T22:53:15.434 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout cluster: 2024-03-11T22:53:15.434 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout id: fa913b96-dff9-11ee-95c8-87774f69a715 2024-03-11T22:53:15.434 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-11T22:53:15.435 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-03-11T22:53:15.435 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout services: 2024-03-11T22:53:15.435 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi153 (age 0.597621s) 2024-03-11T22:53:15.435 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-11T22:53:15.435 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-11T22:53:15.435 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-03-11T22:53:15.435 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout data: 2024-03-11T22:53:15.435 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-11T22:53:15.435 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-11T22:53:15.435 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-11T22:53:15.435 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout pgs: 2024-03-11T22:53:15.435 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-03-11T22:53:15.435 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.788+0000 7fe967c1d700 1 Processor -- start 2024-03-11T22:53:15.435 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.788+0000 7fe967c1d700 1 -- start start 2024-03-11T22:53:15.436 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.788+0000 7fe967c1d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe960071480 0x7fe960105380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:15.436 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.788+0000 7fe967c1d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9601058c0 con 0x7fe960071480 2024-03-11T22:53:15.436 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.789+0000 7fe9659b9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe960071480 0x7fe960105380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:15.436 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.789+0000 7fe9659b9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe960071480 0x7fe960105380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52132/0 (socket says 172.21.15.153:52132) 2024-03-11T22:53:15.436 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.789+0000 7fe9659b9700 1 -- 172.21.15.153:0/3438672962 learned_addr learned my addr 172.21.15.153:0/3438672962 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:53:15.436 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.789+0000 7fe9659b9700 1 -- 172.21.15.153:0/3438672962 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe960105a00 con 0x7fe960071480 2024-03-11T22:53:15.436 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.789+0000 7fe9659b9700 1 --2- 172.21.15.153:0/3438672962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe960071480 0x7fe960105380 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe9500098e0 tx=0x7fe950009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc502e36b6d6f360 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:15.436 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.790+0000 7fe9649b7700 1 -- 172.21.15.153:0/3438672962 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe95000b860 con 0x7fe960071480 2024-03-11T22:53:15.436 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.790+0000 7fe9649b7700 1 -- 172.21.15.153:0/3438672962 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe95000b9c0 con 0x7fe960071480 2024-03-11T22:53:15.436 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.790+0000 7fe9649b7700 1 -- 172.21.15.153:0/3438672962 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe95000bc90 con 0x7fe960071480 2024-03-11T22:53:15.436 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.790+0000 7fe967c1d700 1 -- 172.21.15.153:0/3438672962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe960071480 msgr2=0x7fe960105380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:15.436 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.790+0000 7fe967c1d700 1 --2- 172.21.15.153:0/3438672962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe960071480 0x7fe960105380 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe9500098e0 tx=0x7fe950009250 comp rx=0 tx=0).stop 2024-03-11T22:53:15.436 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.790+0000 7fe967c1d700 1 -- 172.21.15.153:0/3438672962 shutdown_connections 2024-03-11T22:53:15.436 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.790+0000 7fe967c1d700 1 --2- 172.21.15.153:0/3438672962 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe960071480 0x7fe960105380 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:15.436 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.790+0000 7fe967c1d700 1 -- 172.21.15.153:0/3438672962 >> 172.21.15.153:0/3438672962 conn(0x7fe9600fcdd0 msgr2=0x7fe9600ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:53:15.437 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.790+0000 7fe967c1d700 1 -- 172.21.15.153:0/3438672962 shutdown_connections 2024-03-11T22:53:15.437 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.790+0000 7fe967c1d700 1 -- 172.21.15.153:0/3438672962 wait complete. 2024-03-11T22:53:15.437 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.791+0000 7fe967c1d700 1 Processor -- start 2024-03-11T22:53:15.437 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.791+0000 7fe967c1d700 1 -- start start 2024-03-11T22:53:15.437 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.791+0000 7fe967c1d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe960071480 0x7fe960102e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:15.437 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.791+0000 7fe967c1d700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe960101280 con 0x7fe960071480 2024-03-11T22:53:15.437 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.791+0000 7fe9659b9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe960071480 0x7fe960102e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:15.437 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.791+0000 7fe9659b9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe960071480 0x7fe960102e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52144/0 (socket says 172.21.15.153:52144) 2024-03-11T22:53:15.437 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.791+0000 7fe9659b9700 1 -- 172.21.15.153:0/2332429226 learned_addr learned my addr 172.21.15.153:0/2332429226 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:53:15.437 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.791+0000 7fe9659b9700 1 -- 172.21.15.153:0/2332429226 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9500094f0 con 0x7fe960071480 2024-03-11T22:53:15.437 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.791+0000 7fe9659b9700 1 --2- 172.21.15.153:0/2332429226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe960071480 0x7fe960102e00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe950009290 tx=0x7fe950016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:15.437 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.792+0000 7fe9567fc700 1 -- 172.21.15.153:0/2332429226 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe95001c5c0 con 0x7fe960071480 2024-03-11T22:53:15.437 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.792+0000 7fe9567fc700 1 -- 172.21.15.153:0/2332429226 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe95001c740 con 0x7fe960071480 2024-03-11T22:53:15.437 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.792+0000 7fe967c1d700 1 -- 172.21.15.153:0/2332429226 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe960101480 con 0x7fe960071480 2024-03-11T22:53:15.438 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.792+0000 7fe9567fc700 1 -- 172.21.15.153:0/2332429226 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe95001ca50 con 0x7fe960071480 2024-03-11T22:53:15.438 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.792+0000 7fe967c1d700 1 -- 172.21.15.153:0/2332429226 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe960101910 con 0x7fe960071480 2024-03-11T22:53:15.438 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.792+0000 7fe9567fc700 1 -- 172.21.15.153:0/2332429226 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fe95001a450 con 0x7fe960071480 2024-03-11T22:53:15.438 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.792+0000 7fe9567fc700 1 -- 172.21.15.153:0/2332429226 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fe95002e790 con 0x7fe960071480 2024-03-11T22:53:15.438 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.793+0000 7fe967c1d700 1 -- 172.21.15.153:0/2332429226 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe96010ab40 con 0x7fe960071480 2024-03-11T22:53:15.438 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.795+0000 7fe9567fc700 1 -- 172.21.15.153:0/2332429226 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fe95001ce30 con 0x7fe960071480 2024-03-11T22:53:15.438 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.864+0000 7fe967c1d700 1 -- 172.21.15.153:0/2332429226 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fe96002b2a0 con 0x7fe960071480 2024-03-11T22:53:15.438 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.865+0000 7fe9567fc700 1 -- 172.21.15.153:0/2332429226 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7fe95001a840 con 0x7fe960071480 2024-03-11T22:53:15.438 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.865+0000 7fe967c1d700 1 -- 172.21.15.153:0/2332429226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe960071480 msgr2=0x7fe960102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:15.438 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.865+0000 7fe967c1d700 1 --2- 172.21.15.153:0/2332429226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe960071480 0x7fe960102e00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe950009290 tx=0x7fe950016000 comp rx=0 tx=0).stop 2024-03-11T22:53:15.438 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.865+0000 7fe967c1d700 1 -- 172.21.15.153:0/2332429226 shutdown_connections 2024-03-11T22:53:15.438 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.865+0000 7fe967c1d700 1 --2- 172.21.15.153:0/2332429226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe960071480 0x7fe960102e00 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:15.438 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.865+0000 7fe967c1d700 1 -- 172.21.15.153:0/2332429226 >> 172.21.15.153:0/2332429226 conn(0x7fe9600fcdd0 msgr2=0x7fe9600fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:53:15.438 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.865+0000 7fe967c1d700 1 -- 172.21.15.153:0/2332429226 shutdown_connections 2024-03-11T22:53:15.439 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:14.865+0000 7fe967c1d700 1 -- 172.21.15.153:0/2332429226 wait complete. 2024-03-11T22:53:15.439 INFO:teuthology.orchestra.run.smithi153.stdout:mon is available 2024-03-11T22:53:15.439 INFO:teuthology.orchestra.run.smithi153.stdout:Assimilating anything we can from ceph.conf... 2024-03-11T22:53:16.492 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-03-11T22:53:16.492 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout [global] 2024-03-11T22:53:16.492 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout fsid = fa913b96-dff9-11ee-95c8-87774f69a715 2024-03-11T22:53:16.492 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.153:3300,v1:172.21.15.153:6789] 2024-03-11T22:53:16.492 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-11T22:53:16.492 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-11T22:53:16.492 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-11T22:53:16.492 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-11T22:53:16.492 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-03-11T22:53:16.493 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-11T22:53:16.493 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-11T22:53:16.493 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-03-11T22:53:16.493 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout [osd] 2024-03-11T22:53:16.493 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-11T22:53:16.493 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-11T22:53:16.493 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-11T22:53:16.493 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-11T22:53:16.493 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-11T22:53:16.493 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.868+0000 7f7708933700 1 Processor -- start 2024-03-11T22:53:16.493 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.868+0000 7f7708933700 1 -- start start 2024-03-11T22:53:16.493 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.868+0000 7f7708933700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7704103d80 0x7f7704104190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:16.493 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.868+0000 7f7708933700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77041046d0 con 0x7f7704103d80 2024-03-11T22:53:16.493 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.868+0000 7f770259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7704103d80 0x7f7704104190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:16.493 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.868+0000 7f770259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7704103d80 0x7f7704104190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52150/0 (socket says 172.21.15.153:52150) 2024-03-11T22:53:16.493 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.868+0000 7f770259c700 1 -- 172.21.15.153:0/3902248768 learned_addr learned my addr 172.21.15.153:0/3902248768 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:53:16.494 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.868+0000 7f770259c700 1 -- 172.21.15.153:0/3902248768 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7704104810 con 0x7f7704103d80 2024-03-11T22:53:16.494 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.869+0000 7f770259c700 1 --2- 172.21.15.153:0/3902248768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7704103d80 0x7f7704104190 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f76f40098e0 tx=0x7f76f4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76a082b7ed4bb31d server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:16.494 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.869+0000 7f770159a700 1 -- 172.21.15.153:0/3902248768 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76f400b860 con 0x7f7704103d80 2024-03-11T22:53:16.494 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.869+0000 7f770159a700 1 -- 172.21.15.153:0/3902248768 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76f400b9c0 con 0x7f7704103d80 2024-03-11T22:53:16.494 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.869+0000 7f770159a700 1 -- 172.21.15.153:0/3902248768 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76f400bc90 con 0x7f7704103d80 2024-03-11T22:53:16.494 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.869+0000 7f7708933700 1 -- 172.21.15.153:0/3902248768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7704103d80 msgr2=0x7f7704104190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:16.494 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.869+0000 7f7708933700 1 --2- 172.21.15.153:0/3902248768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7704103d80 0x7f7704104190 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f76f40098e0 tx=0x7f76f4009250 comp rx=0 tx=0).stop 2024-03-11T22:53:16.494 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.869+0000 7f7708933700 1 -- 172.21.15.153:0/3902248768 shutdown_connections 2024-03-11T22:53:16.494 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.869+0000 7f7708933700 1 --2- 172.21.15.153:0/3902248768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7704103d80 0x7f7704104190 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:16.494 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.869+0000 7f7708933700 1 -- 172.21.15.153:0/3902248768 >> 172.21.15.153:0/3902248768 conn(0x7f77040fc580 msgr2=0x7f77040fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:53:16.494 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.869+0000 7f7708933700 1 -- 172.21.15.153:0/3902248768 shutdown_connections 2024-03-11T22:53:16.494 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.870+0000 7f7708933700 1 -- 172.21.15.153:0/3902248768 wait complete. 2024-03-11T22:53:16.494 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.870+0000 7f7708933700 1 Processor -- start 2024-03-11T22:53:16.494 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.870+0000 7f7708933700 1 -- start start 2024-03-11T22:53:16.495 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.870+0000 7f7708933700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7704103d80 0x7f7704111060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:16.495 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.870+0000 7f7708933700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77041115a0 con 0x7f7704103d80 2024-03-11T22:53:16.495 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.870+0000 7f770259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7704103d80 0x7f7704111060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:16.495 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.870+0000 7f770259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7704103d80 0x7f7704111060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52166/0 (socket says 172.21.15.153:52166) 2024-03-11T22:53:16.495 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.870+0000 7f770259c700 1 -- 172.21.15.153:0/3403694774 learned_addr learned my addr 172.21.15.153:0/3403694774 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:53:16.495 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.871+0000 7f770259c700 1 -- 172.21.15.153:0/3403694774 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f40094f0 con 0x7f7704103d80 2024-03-11T22:53:16.495 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.871+0000 7f770259c700 1 --2- 172.21.15.153:0/3403694774 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7704103d80 0x7f7704111060 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f76f4003080 tx=0x7f76f4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:16.495 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.871+0000 7f76f2ffd700 1 -- 172.21.15.153:0/3403694774 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76f401c5c0 con 0x7f7704103d80 2024-03-11T22:53:16.495 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.871+0000 7f76f2ffd700 1 -- 172.21.15.153:0/3403694774 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76f401c720 con 0x7f7704103d80 2024-03-11T22:53:16.495 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.871+0000 7f76f2ffd700 1 -- 172.21.15.153:0/3403694774 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76f401ca30 con 0x7f7704103d80 2024-03-11T22:53:16.495 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.871+0000 7f7708933700 1 -- 172.21.15.153:0/3403694774 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77041117a0 con 0x7f7704103d80 2024-03-11T22:53:16.495 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.871+0000 7f7708933700 1 -- 172.21.15.153:0/3403694774 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7704112440 con 0x7f7704103d80 2024-03-11T22:53:16.495 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.871+0000 7f76f2ffd700 1 -- 172.21.15.153:0/3403694774 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f76f401a450 con 0x7f7704103d80 2024-03-11T22:53:16.495 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.871+0000 7f76f2ffd700 1 -- 172.21.15.153:0/3403694774 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f76f402e790 con 0x7f7704103d80 2024-03-11T22:53:16.496 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.872+0000 7f7708933700 1 -- 172.21.15.153:0/3403694774 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f770405f5c0 con 0x7f7704103d80 2024-03-11T22:53:16.496 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.874+0000 7f76f2ffd700 1 -- 172.21.15.153:0/3403694774 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f76f401a890 con 0x7f7704103d80 2024-03-11T22:53:16.496 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.939+0000 7f7708933700 1 -- 172.21.15.153:0/3403694774 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f770410acf0 con 0x7f7704103d80 2024-03-11T22:53:16.496 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.943+0000 7f76f2ffd700 1 -- 172.21.15.153:0/3403694774 <== mon.0 v2:172.21.15.153:3300/0 7 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f76f402ee80 con 0x7f7704103d80 2024-03-11T22:53:16.496 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.943+0000 7f76f2ffd700 1 -- 172.21.15.153:0/3403694774 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+431 (secure 0 0 0) 0x7f76f4037590 con 0x7f7704103d80 2024-03-11T22:53:16.496 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.944+0000 7f7708933700 1 -- 172.21.15.153:0/3403694774 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7704103d80 msgr2=0x7f7704111060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:16.496 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.944+0000 7f7708933700 1 --2- 172.21.15.153:0/3403694774 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7704103d80 0x7f7704111060 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f76f4003080 tx=0x7f76f4016000 comp rx=0 tx=0).stop 2024-03-11T22:53:16.496 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.944+0000 7f7708933700 1 -- 172.21.15.153:0/3403694774 shutdown_connections 2024-03-11T22:53:16.496 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.944+0000 7f7708933700 1 --2- 172.21.15.153:0/3403694774 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7704103d80 0x7f7704111060 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:16.496 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.944+0000 7f7708933700 1 -- 172.21.15.153:0/3403694774 >> 172.21.15.153:0/3403694774 conn(0x7f77040fc580 msgr2=0x7f7704107e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:53:16.496 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.944+0000 7f7708933700 1 -- 172.21.15.153:0/3403694774 shutdown_connections 2024-03-11T22:53:16.496 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:15.944+0000 7f7708933700 1 -- 172.21.15.153:0/3403694774 wait complete. 2024-03-11T22:53:16.496 INFO:teuthology.orchestra.run.smithi153.stdout:Generating new minimal ceph.conf... 2024-03-11T22:53:16.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:16 smithi153 ceph-mon[28589]: from='client.? 172.21.15.153:0/3403694774' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-11T22:53:16.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:16 smithi153 ceph-mon[28589]: from='client.? 172.21.15.153:0/3403694774' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-11T22:53:17.522 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.909+0000 7f6ac10f5700 1 Processor -- start 2024-03-11T22:53:17.522 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.909+0000 7f6ac10f5700 1 -- start start 2024-03-11T22:53:17.522 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.910+0000 7f6ac10f5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6abc102d90 0x7f6abc1031a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:17.522 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.910+0000 7f6ac10f5700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc1036e0 con 0x7f6abc102d90 2024-03-11T22:53:17.522 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.910+0000 7f6abad9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6abc102d90 0x7f6abc1031a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:17.522 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.910+0000 7f6abad9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6abc102d90 0x7f6abc1031a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52174/0 (socket says 172.21.15.153:52174) 2024-03-11T22:53:17.522 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.910+0000 7f6abad9d700 1 -- 172.21.15.153:0/380449067 learned_addr learned my addr 172.21.15.153:0/380449067 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:53:17.522 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.910+0000 7f6abad9d700 1 -- 172.21.15.153:0/380449067 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6abc103820 con 0x7f6abc102d90 2024-03-11T22:53:17.522 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.910+0000 7f6abad9d700 1 --2- 172.21.15.153:0/380449067 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6abc102d90 0x7f6abc1031a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6ab00098e0 tx=0x7f6ab0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0d2dd8de456258e server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:17.523 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.910+0000 7f6ab9d9b700 1 -- 172.21.15.153:0/380449067 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ab000b860 con 0x7f6abc102d90 2024-03-11T22:53:17.523 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.910+0000 7f6ab9d9b700 1 -- 172.21.15.153:0/380449067 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f6ab000be50 con 0x7f6abc102d90 2024-03-11T22:53:17.523 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.911+0000 7f6ab9d9b700 1 -- 172.21.15.153:0/380449067 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ab0010380 con 0x7f6abc102d90 2024-03-11T22:53:17.523 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.911+0000 7f6ac10f5700 1 -- 172.21.15.153:0/380449067 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6abc102d90 msgr2=0x7f6abc1031a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:17.523 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.911+0000 7f6ac10f5700 1 --2- 172.21.15.153:0/380449067 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6abc102d90 0x7f6abc1031a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6ab00098e0 tx=0x7f6ab0009250 comp rx=0 tx=0).stop 2024-03-11T22:53:17.523 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.911+0000 7f6ac10f5700 1 -- 172.21.15.153:0/380449067 shutdown_connections 2024-03-11T22:53:17.523 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.911+0000 7f6ac10f5700 1 --2- 172.21.15.153:0/380449067 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6abc102d90 0x7f6abc1031a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:17.523 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.911+0000 7f6ac10f5700 1 -- 172.21.15.153:0/380449067 >> 172.21.15.153:0/380449067 conn(0x7f6abc0fc580 msgr2=0x7f6abc0fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:53:17.523 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.911+0000 7f6ac10f5700 1 -- 172.21.15.153:0/380449067 shutdown_connections 2024-03-11T22:53:17.523 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.911+0000 7f6ac10f5700 1 -- 172.21.15.153:0/380449067 wait complete. 2024-03-11T22:53:17.523 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.912+0000 7f6ac10f5700 1 Processor -- start 2024-03-11T22:53:17.523 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.912+0000 7f6ac10f5700 1 -- start start 2024-03-11T22:53:17.523 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.912+0000 7f6ac10f5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6abc102d90 0x7f6abc1139f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:17.523 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.912+0000 7f6ac10f5700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc113f30 con 0x7f6abc102d90 2024-03-11T22:53:17.524 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.912+0000 7f6abad9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6abc102d90 0x7f6abc1139f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:17.524 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.912+0000 7f6abad9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6abc102d90 0x7f6abc1139f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52184/0 (socket says 172.21.15.153:52184) 2024-03-11T22:53:17.524 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.912+0000 7f6abad9d700 1 -- 172.21.15.153:0/2348766143 learned_addr learned my addr 172.21.15.153:0/2348766143 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:53:17.524 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.912+0000 7f6abad9d700 1 -- 172.21.15.153:0/2348766143 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab00094f0 con 0x7f6abc102d90 2024-03-11T22:53:17.524 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.912+0000 7f6abad9d700 1 --2- 172.21.15.153:0/2348766143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6abc102d90 0x7f6abc1139f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6ab000b440 tx=0x7f6ab0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:17.524 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.913+0000 7f6aa37fe700 1 -- 172.21.15.153:0/2348766143 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ab00107c0 con 0x7f6abc102d90 2024-03-11T22:53:17.524 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.913+0000 7f6aa37fe700 1 -- 172.21.15.153:0/2348766143 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f6ab0010db0 con 0x7f6abc102d90 2024-03-11T22:53:17.524 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.913+0000 7f6ac10f5700 1 -- 172.21.15.153:0/2348766143 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6abc114130 con 0x7f6abc102d90 2024-03-11T22:53:17.524 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.913+0000 7f6aa37fe700 1 -- 172.21.15.153:0/2348766143 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ab001ad80 con 0x7f6abc102d90 2024-03-11T22:53:17.524 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.913+0000 7f6ac10f5700 1 -- 172.21.15.153:0/2348766143 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6abc114dd0 con 0x7f6abc102d90 2024-03-11T22:53:17.524 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.913+0000 7f6aa37fe700 1 -- 172.21.15.153:0/2348766143 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f6ab002e6c0 con 0x7f6abc102d90 2024-03-11T22:53:17.524 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.913+0000 7f6aa37fe700 1 -- 172.21.15.153:0/2348766143 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f6ab00249f0 con 0x7f6abc102d90 2024-03-11T22:53:17.525 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.914+0000 7f6ac10f5700 1 -- 172.21.15.153:0/2348766143 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6abc05f5c0 con 0x7f6abc102d90 2024-03-11T22:53:17.525 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.916+0000 7f6aa37fe700 1 -- 172.21.15.153:0/2348766143 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f6ab0010920 con 0x7f6abc102d90 2024-03-11T22:53:17.525 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.981+0000 7f6ac10f5700 1 -- 172.21.15.153:0/2348766143 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f6abc064ad0 con 0x7f6abc102d90 2024-03-11T22:53:17.525 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.981+0000 7f6aa37fe700 1 -- 172.21.15.153:0/2348766143 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f6ab0010ac0 con 0x7f6abc102d90 2024-03-11T22:53:17.525 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.982+0000 7f6ac10f5700 1 -- 172.21.15.153:0/2348766143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6abc102d90 msgr2=0x7f6abc1139f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:17.525 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.982+0000 7f6ac10f5700 1 --2- 172.21.15.153:0/2348766143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6abc102d90 0x7f6abc1139f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6ab000b440 tx=0x7f6ab0009250 comp rx=0 tx=0).stop 2024-03-11T22:53:17.525 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.982+0000 7f6ac10f5700 1 -- 172.21.15.153:0/2348766143 shutdown_connections 2024-03-11T22:53:17.525 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.982+0000 7f6ac10f5700 1 --2- 172.21.15.153:0/2348766143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6abc102d90 0x7f6abc1139f0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:17.525 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.982+0000 7f6ac10f5700 1 -- 172.21.15.153:0/2348766143 >> 172.21.15.153:0/2348766143 conn(0x7f6abc0fc580 msgr2=0x7f6abc0fd450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:53:17.525 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.982+0000 7f6ac10f5700 1 -- 172.21.15.153:0/2348766143 shutdown_connections 2024-03-11T22:53:17.525 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:16.982+0000 7f6ac10f5700 1 -- 172.21.15.153:0/2348766143 wait complete. 2024-03-11T22:53:17.525 INFO:teuthology.orchestra.run.smithi153.stdout:Restarting the monitor... 2024-03-11T22:53:17.545 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:17 smithi153 ceph-mon[28589]: from='client.? 172.21.15.153:0/2348766143' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:53:17.810 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:17 smithi153 systemd[1]: Stopping Ceph mon.smithi153 for fa913b96-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:53:18.259 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28585]: 2024-03-11T22:53:17.805+0000 7fef1c31b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi153 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:53:18.260 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28585]: 2024-03-11T22:53:17.805+0000 7fef1c31b700 -1 mon.smithi153@0(leader) e1 *** Got Signal Terminated *** 2024-03-11T22:53:18.260 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:18 smithi153 podman[28833]: 2024-03-11 22:53:17.999772747 +0000 UTC m=+0.318925136 container died 315e8e0efa43fddbeeba14a84a23634582032f161715565aafd337e14a595299 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153, RELEASE=quincy-c186d0c, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:53:18.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:18 smithi153 podman[28833]: 2024-03-11 22:53:18.346931754 +0000 UTC m=+0.666084141 container cleanup 315e8e0efa43fddbeeba14a84a23634582032f161715565aafd337e14a595299 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=quincy-c186d0c, org.label-schema.license=GPLv2) 2024-03-11T22:53:18.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:18 smithi153 bash[28833]: ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153 2024-03-11T22:53:18.969 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:18 smithi153 podman[28847]: 2024-03-11 22:53:18.614383755 +0000 UTC m=+0.611334160 container remove 315e8e0efa43fddbeeba14a84a23634582032f161715565aafd337e14a595299 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=quincy-c186d0c) 2024-03-11T22:53:18.970 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:18 smithi153 systemd[1]: ceph-fa913b96-dff9-11ee-95c8-87774f69a715@mon.smithi153.service: Deactivated successfully. 2024-03-11T22:53:18.970 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:18 smithi153 systemd[1]: Stopped Ceph mon.smithi153 for fa913b96-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:53:18.970 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:18 smithi153 systemd[1]: Starting Ceph mon.smithi153 for fa913b96-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:53:19.220 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:18 smithi153 podman[28943]: 2024-03-11 22:53:18.965514146 +0000 UTC m=+0.113814413 container create 0dad3ee1fdfffa6fa2e3579fca46d272ad782cee39b501b8a50a1c6a88af8bdf (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153, RELEASE=quincy-c186d0c, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-11T22:53:19.220 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:18 smithi153 podman[28943]: 2024-03-11 22:53:18.869053992 +0000 UTC m=+0.017354278 image pull a92afbb1c2834deb6f1bddda3f3c554061dd3ec88e0495c17c1373d9018e466b quay.ceph.io/ceph-ci/ceph:quincy 2024-03-11T22:53:19.220 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 podman[28943]: 2024-03-11 22:53:19.184643357 +0000 UTC m=+0.332943623 container init 0dad3ee1fdfffa6fa2e3579fca46d272ad782cee39b501b8a50a1c6a88af8bdf (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-c186d0c, io.buildah.version=1.33.5) 2024-03-11T22:53:19.220 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 podman[28943]: 2024-03-11 22:53:19.187998661 +0000 UTC m=+0.336298937 container start 0dad3ee1fdfffa6fa2e3579fca46d272ad782cee39b501b8a50a1c6a88af8bdf (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , RELEASE=quincy-c186d0c, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:53:19.220 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T22:53:19.220 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: ceph version 17.2.7-620-gc186d0ca (c186d0ca78be18052defec2e0c064f7869af6d3f) quincy (stable), process ceph-mon, pid 2 2024-03-11T22:53:19.221 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: pidfile_write: ignore empty --pid-file 2024-03-11T22:53:19.221 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: load: jerasure load: lrc 2024-03-11T22:53:19.221 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: RocksDB version: 6.15.5 2024-03-11T22:53:19.221 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-03-11T22:53:19.221 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Compile date Mar 11 2024 2024-03-11T22:53:19.221 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: DB SUMMARY 2024-03-11T22:53:19.221 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: DB Session ID: Q0GB0TZBH1QLRYZ9D3VZ 2024-03-11T22:53:19.221 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: CURRENT file: CURRENT 2024-03-11T22:53:19.221 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T22:53:19.221 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-03-11T22:53:19.221 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi153/store.db dir, Total Num: 1, files: 000008.sst 2024-03-11T22:53:19.221 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi153/store.db: 000010.log size: 97310 ; 2024-03-11T22:53:19.221 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.error_if_exists: 0 2024-03-11T22:53:19.221 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.create_if_missing: 0 2024-03-11T22:53:19.222 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.paranoid_checks: 1 2024-03-11T22:53:19.222 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T22:53:19.222 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.env: 0x557dede4ba80 2024-03-11T22:53:19.222 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.fs: Posix File System 2024-03-11T22:53:19.222 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.info_log: 0x557deea57cc0 2024-03-11T22:53:19.222 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T22:53:19.222 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.statistics: (nil) 2024-03-11T22:53:19.222 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.use_fsync: 0 2024-03-11T22:53:19.222 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.max_log_file_size: 0 2024-03-11T22:53:19.222 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T22:53:19.222 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T22:53:19.222 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T22:53:19.222 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T22:53:19.222 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.allow_fallocate: 1 2024-03-11T22:53:19.223 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T22:53:19.223 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T22:53:19.223 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.use_direct_reads: 0 2024-03-11T22:53:19.223 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T22:53:19.223 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T22:53:19.223 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.db_log_dir: 2024-03-11T22:53:19.223 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi153/store.db 2024-03-11T22:53:19.223 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T22:53:19.223 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T22:53:19.223 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T22:53:19.223 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T22:53:19.223 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T22:53:19.223 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T22:53:19.223 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T22:53:19.223 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T22:53:19.224 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.write_buffer_manager: 0x557def7ccae0 2024-03-11T22:53:19.224 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T22:53:19.224 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-03-11T22:53:19.224 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T22:53:19.224 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T22:53:19.224 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T22:53:19.224 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T22:53:19.224 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T22:53:19.224 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T22:53:19.224 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T22:53:19.225 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.unordered_write: 0 2024-03-11T22:53:19.225 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T22:53:19.225 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T22:53:19.225 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T22:53:19.225 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T22:53:19.225 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.row_cache: None 2024-03-11T22:53:19.225 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.wal_filter: None 2024-03-11T22:53:19.225 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T22:53:19.225 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T22:53:19.225 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.preserve_deletes: 0 2024-03-11T22:53:19.225 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.two_write_queues: 0 2024-03-11T22:53:19.225 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T22:53:19.226 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.atomic_flush: 0 2024-03-11T22:53:19.226 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T22:53:19.226 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T22:53:19.226 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T22:53:19.226 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.log_readahead_size: 0 2024-03-11T22:53:19.226 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T22:53:19.226 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T22:53:19.226 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T22:53:19.226 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T22:53:19.226 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T22:53:19.226 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T22:53:19.226 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.max_background_jobs: 2 2024-03-11T22:53:19.226 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.max_background_compactions: -1 2024-03-11T22:53:19.226 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.max_subcompactions: 1 2024-03-11T22:53:19.226 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T22:53:19.227 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T22:53:19.227 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T22:53:19.227 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T22:53:19.227 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T22:53:19.227 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T22:53:19.227 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T22:53:19.227 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T22:53:19.227 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.max_open_files: -1 2024-03-11T22:53:19.227 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T22:53:19.227 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T22:53:19.227 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T22:53:19.227 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T22:53:19.227 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.max_background_flushes: -1 2024-03-11T22:53:19.227 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Compression algorithms supported: 2024-03-11T22:53:19.227 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T22:53:19.227 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: kZSTD supported: 0 2024-03-11T22:53:19.227 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: kXpressCompression supported: 0 2024-03-11T22:53:19.228 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T22:53:19.228 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: kZlibCompression supported: 1 2024-03-11T22:53:19.228 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: kSnappyCompression supported: 1 2024-03-11T22:53:19.228 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: kLZ4Compression supported: 1 2024-03-11T22:53:19.228 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: kBZip2Compression supported: 0 2024-03-11T22:53:19.228 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T22:53:19.228 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi153/store.db/MANIFEST-000009 2024-03-11T22:53:19.228 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-03-11T22:53:19.228 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T22:53:19.228 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.merge_operator: 2024-03-11T22:53:19.228 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.compaction_filter: None 2024-03-11T22:53:19.228 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T22:53:19.228 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T22:53:19.228 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T22:53:19.228 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T22:53:19.229 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x557deeaacf58) 2024-03-11T22:53:19.229 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: cache_index_and_filter_blocks: 1 2024-03-11T22:53:19.229 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T22:53:19.229 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T22:53:19.229 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: pin_top_level_index_and_filter: 1 2024-03-11T22:53:19.229 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: index_type: 0 2024-03-11T22:53:19.229 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: data_block_index_type: 0 2024-03-11T22:53:19.229 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: index_shortening: 1 2024-03-11T22:53:19.229 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T22:53:19.229 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: hash_index_allow_collision: 1 2024-03-11T22:53:19.229 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: checksum: 1 2024-03-11T22:53:19.229 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: no_block_cache: 0 2024-03-11T22:53:19.229 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_cache: 0x557deeadf090 2024-03-11T22:53:19.229 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_cache_name: BinnedLRUCache 2024-03-11T22:53:19.229 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_cache_options: 2024-03-11T22:53:19.229 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: capacity : 536870912 2024-03-11T22:53:19.230 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: num_shard_bits : 4 2024-03-11T22:53:19.230 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: strict_capacity_limit : 0 2024-03-11T22:53:19.230 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: high_pri_pool_ratio: 0.000 2024-03-11T22:53:19.230 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_cache_compressed: (nil) 2024-03-11T22:53:19.230 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: persistent_cache: (nil) 2024-03-11T22:53:19.230 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_size: 4096 2024-03-11T22:53:19.230 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_size_deviation: 10 2024-03-11T22:53:19.230 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_restart_interval: 16 2024-03-11T22:53:19.230 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: index_block_restart_interval: 1 2024-03-11T22:53:19.230 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: metadata_block_size: 4096 2024-03-11T22:53:19.230 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: partition_filters: 0 2024-03-11T22:53:19.230 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: use_delta_encoding: 1 2024-03-11T22:53:19.230 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-03-11T22:53:19.230 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: whole_key_filtering: 1 2024-03-11T22:53:19.231 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: verify_compression: 0 2024-03-11T22:53:19.231 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: read_amp_bytes_per_bit: 0 2024-03-11T22:53:19.231 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: format_version: 4 2024-03-11T22:53:19.231 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: enable_index_compression: 1 2024-03-11T22:53:19.231 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_align: 0 2024-03-11T22:53:19.231 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T22:53:19.231 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T22:53:19.231 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.compression: NoCompression 2024-03-11T22:53:19.231 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T22:53:19.231 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T22:53:19.231 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T22:53:19.231 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.num_levels: 7 2024-03-11T22:53:19.231 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T22:53:19.295 INFO:teuthology.orchestra.run.smithi153.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-11T22:53:19.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T22:53:19.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T22:53:19.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T22:53:19.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T22:53:19.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T22:53:19.533 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T22:53:19.533 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T22:53:19.533 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T22:53:19.533 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T22:53:19.533 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T22:53:19.533 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T22:53:19.533 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T22:53:19.533 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T22:53:19.533 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T22:53:19.533 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T22:53:19.533 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T22:53:19.533 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T22:53:19.533 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T22:53:19.533 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T22:53:19.533 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T22:53:19.534 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T22:53:19.534 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T22:53:19.534 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T22:53:19.534 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T22:53:19.534 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T22:53:19.534 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T22:53:19.534 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T22:53:19.534 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T22:53:19.534 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T22:53:19.534 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T22:53:19.534 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T22:53:19.534 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T22:53:19.534 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T22:53:19.534 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.arena_block_size: 4194304 2024-03-11T22:53:19.534 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T22:53:19.535 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T22:53:19.535 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-03-11T22:53:19.535 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T22:53:19.535 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T22:53:19.535 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T22:53:19.535 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T22:53:19.535 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T22:53:19.535 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T22:53:19.535 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T22:53:19.535 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T22:53:19.535 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T22:53:19.535 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T22:53:19.535 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T22:53:19.535 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.table_properties_collectors: 2024-03-11T22:53:19.535 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.inplace_update_support: 0 2024-03-11T22:53:19.536 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T22:53:19.536 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T22:53:19.536 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T22:53:19.536 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T22:53:19.536 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.bloom_locality: 0 2024-03-11T22:53:19.536 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.max_successive_merges: 0 2024-03-11T22:53:19.536 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T22:53:19.536 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T22:53:19.536 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T22:53:19.536 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T22:53:19.536 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.ttl: 2592000 2024-03-11T22:53:19.536 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T22:53:19.536 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.enable_blob_files: false 2024-03-11T22:53:19.536 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.min_blob_size: 0 2024-03-11T22:53:19.536 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T22:53:19.537 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T22:53:19.537 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T22:53:19.537 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T22:53:19.537 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi153/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-11T22:53:19.537 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-03-11T22:53:19.537 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-03-11T22:53:19.537 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710197599215265, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-03-11T22:53:19.537 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-03-11T22:53:19.537 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-03-11T22:53:19.537 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710197599216381, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 92582, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 90677, "index_size": 265, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13356, "raw_average_key_size": 49, "raw_value_size": 83392, "raw_average_value_size": 310, "num_data_blocks": 11, "num_entries": 269, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1710197599, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "8ad2ba10-bcb7-4a31-8657-571db258ed75", "db_session_id": "Q0GB0TZBH1QLRYZ9D3VZ"}} 2024-03-11T22:53:19.537 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-03-11T22:53:19.538 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710197599216823, "job": 1, "event": "recovery_finished"} 2024-03-11T22:53:19.538 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi153/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T22:53:19.538 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x557deea73180 2024-03-11T22:53:19.538 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: DB pointer 0x557deeb48000 2024-03-11T22:53:19.538 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-03-11T22:53:19.538 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-03-11T22:53:19.538 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** DB Stats ** 2024-03-11T22:53:19.538 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T22:53:19.538 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T22:53:19.538 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T22:53:19.538 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T22:53:19.538 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T22:53:19.538 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-03-11T22:53:19.539 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T22:53:19.539 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-03-11T22:53:19.539 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** Compaction Stats [default] ** 2024-03-11T22:53:19.539 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-11T22:53:19.539 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T22:53:19.539 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: L0 2/0 92.21 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 117.3 0.00 0.00 1 0.001 0 0 2024-03-11T22:53:19.539 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Sum 2/0 92.21 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 117.3 0.00 0.00 1 0.001 0 0 2024-03-11T22:53:19.539 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 117.3 0.00 0.00 1 0.001 0 0 2024-03-11T22:53:19.539 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-03-11T22:53:19.539 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** Compaction Stats [default] ** 2024-03-11T22:53:19.539 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-11T22:53:19.539 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T22:53:19.539 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 117.3 0.00 0.00 1 0.001 0 0 2024-03-11T22:53:19.539 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T22:53:19.539 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T22:53:19.540 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T22:53:19.540 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T22:53:19.540 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T22:53:19.540 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T22:53:19.540 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Cumulative compaction: 0.00 GB write, 21.35 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:53:19.540 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Interval compaction: 0.00 GB write, 21.35 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:53:19.540 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T22:53:19.540 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-03-11T22:53:19.540 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T22:53:19.540 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-03-11T22:53:19.540 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** Compaction Stats [default] ** 2024-03-11T22:53:19.540 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-11T22:53:19.540 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T22:53:19.540 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: L0 2/0 92.21 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 117.3 0.00 0.00 1 0.001 0 0 2024-03-11T22:53:19.540 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Sum 2/0 92.21 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 117.3 0.00 0.00 1 0.001 0 0 2024-03-11T22:53:19.540 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-03-11T22:53:19.541 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-03-11T22:53:19.541 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** Compaction Stats [default] ** 2024-03-11T22:53:19.541 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-11T22:53:19.541 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T22:53:19.541 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 117.3 0.00 0.00 1 0.001 0 0 2024-03-11T22:53:19.541 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T22:53:19.541 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T22:53:19.541 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T22:53:19.541 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T22:53:19.541 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T22:53:19.541 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T22:53:19.541 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Cumulative compaction: 0.00 GB write, 21.19 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:53:19.541 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:53:19.541 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T22:53:19.541 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-03-11T22:53:19.542 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T22:53:19.542 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: starting mon.smithi153 rank 0 at public addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] at bind addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi153 fsid fa913b96-dff9-11ee-95c8-87774f69a715 2024-03-11T22:53:19.542 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: mon.smithi153@-1(???) e1 preinit fsid fa913b96-dff9-11ee-95c8-87774f69a715 2024-03-11T22:53:19.542 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: mon.smithi153@-1(???).mds e1 new map 2024-03-11T22:53:19.542 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: mon.smithi153@-1(???).mds e1 print_map 2024-03-11T22:53:19.542 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: e1 2024-03-11T22:53:19.542 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T22:53:19.542 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-11T22:53:19.542 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: legacy client fscid: -1 2024-03-11T22:53:19.542 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-03-11T22:53:19.542 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: No filesystems configured 2024-03-11T22:53:19.542 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: mon.smithi153@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T22:53:19.542 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: mon.smithi153@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:53:19.542 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: mon.smithi153@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:53:19.542 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: mon.smithi153@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:53:19.543 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: mon.smithi153@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-11T22:53:19.543 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 bash[28943]: 0dad3ee1fdfffa6fa2e3579fca46d272ad782cee39b501b8a50a1c6a88af8bdf 2024-03-11T22:53:19.543 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 systemd[1]: Started Ceph mon.smithi153 for fa913b96-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:53:19.543 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: mon.smithi153 is new leader, mons smithi153 in quorum (ranks 0) 2024-03-11T22:53:19.543 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: monmap e1: 1 mons at {smithi153=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:53:19.543 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: fsmap 2024-03-11T22:53:19.543 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: osdmap e1: 0 total, 0 up, 0 in 2024-03-11T22:53:19.543 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:19 smithi153 ceph-mon[28962]: mgrmap e1: no daemons active 2024-03-11T22:53:20.297 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.712+0000 7ffb2ffb2700 1 Processor -- start 2024-03-11T22:53:20.302 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.712+0000 7ffb2ffb2700 1 -- start start 2024-03-11T22:53:20.302 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.712+0000 7ffb2ffb2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb28078ce0 0x7ffb280771f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:20.302 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.712+0000 7ffb2ffb2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb280790f0 con 0x7ffb28078ce0 2024-03-11T22:53:20.302 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.713+0000 7ffb2dd4e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb28078ce0 0x7ffb280771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:20.303 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.713+0000 7ffb2dd4e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb28078ce0 0x7ffb280771f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52196/0 (socket says 172.21.15.153:52196) 2024-03-11T22:53:20.303 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.713+0000 7ffb2dd4e700 1 -- 172.21.15.153:0/2085185926 learned_addr learned my addr 172.21.15.153:0/2085185926 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:53:20.303 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.713+0000 7ffb2dd4e700 1 -- 172.21.15.153:0/2085185926 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb28077730 con 0x7ffb28078ce0 2024-03-11T22:53:20.303 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.713+0000 7ffb2dd4e700 1 --2- 172.21.15.153:0/2085185926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb28078ce0 0x7ffb280771f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ffb240098e0 tx=0x7ffb24009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c07632831e06f8c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:20.303 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.713+0000 7ffb2cd4c700 1 -- 172.21.15.153:0/2085185926 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb2400b860 con 0x7ffb28078ce0 2024-03-11T22:53:20.303 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.713+0000 7ffb2cd4c700 1 -- 172.21.15.153:0/2085185926 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ffb2400be50 con 0x7ffb28078ce0 2024-03-11T22:53:20.303 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.713+0000 7ffb2cd4c700 1 -- 172.21.15.153:0/2085185926 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb24010380 con 0x7ffb28078ce0 2024-03-11T22:53:20.303 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.714+0000 7ffb2ffb2700 1 -- 172.21.15.153:0/2085185926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb28078ce0 msgr2=0x7ffb280771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:20.303 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.714+0000 7ffb2ffb2700 1 --2- 172.21.15.153:0/2085185926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb28078ce0 0x7ffb280771f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ffb240098e0 tx=0x7ffb24009250 comp rx=0 tx=0).stop 2024-03-11T22:53:20.303 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.714+0000 7ffb2ffb2700 1 -- 172.21.15.153:0/2085185926 shutdown_connections 2024-03-11T22:53:20.303 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.714+0000 7ffb2ffb2700 1 --2- 172.21.15.153:0/2085185926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb28078ce0 0x7ffb280771f0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:20.303 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.714+0000 7ffb2ffb2700 1 -- 172.21.15.153:0/2085185926 >> 172.21.15.153:0/2085185926 conn(0x7ffb280fded0 msgr2=0x7ffb28100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:53:20.303 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.714+0000 7ffb2ffb2700 1 -- 172.21.15.153:0/2085185926 shutdown_connections 2024-03-11T22:53:20.303 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.714+0000 7ffb2ffb2700 1 -- 172.21.15.153:0/2085185926 wait complete. 2024-03-11T22:53:20.303 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.714+0000 7ffb2ffb2700 1 Processor -- start 2024-03-11T22:53:20.304 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.714+0000 7ffb2ffb2700 1 -- start start 2024-03-11T22:53:20.304 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.715+0000 7ffb2ffb2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb28078ce0 0x7ffb28113820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:20.304 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.715+0000 7ffb2ffb2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb28113d60 con 0x7ffb28078ce0 2024-03-11T22:53:20.304 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.715+0000 7ffb2dd4e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb28078ce0 0x7ffb28113820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:20.304 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.715+0000 7ffb2dd4e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb28078ce0 0x7ffb28113820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52202/0 (socket says 172.21.15.153:52202) 2024-03-11T22:53:20.304 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.715+0000 7ffb2dd4e700 1 -- 172.21.15.153:0/3623780320 learned_addr learned my addr 172.21.15.153:0/3623780320 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:53:20.304 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.715+0000 7ffb2dd4e700 1 -- 172.21.15.153:0/3623780320 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb240094f0 con 0x7ffb28078ce0 2024-03-11T22:53:20.304 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.715+0000 7ffb2dd4e700 1 --2- 172.21.15.153:0/3623780320 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb28078ce0 0x7ffb28113820 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ffb2400b440 tx=0x7ffb24009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:20.304 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.715+0000 7ffb1e7fc700 1 -- 172.21.15.153:0/3623780320 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb24010760 con 0x7ffb28078ce0 2024-03-11T22:53:20.304 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.715+0000 7ffb1e7fc700 1 -- 172.21.15.153:0/3623780320 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ffb24010d50 con 0x7ffb28078ce0 2024-03-11T22:53:20.304 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.715+0000 7ffb1e7fc700 1 -- 172.21.15.153:0/3623780320 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb2401adc0 con 0x7ffb28078ce0 2024-03-11T22:53:20.304 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.715+0000 7ffb2ffb2700 1 -- 172.21.15.153:0/3623780320 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb28113f60 con 0x7ffb28078ce0 2024-03-11T22:53:20.304 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.715+0000 7ffb2ffb2700 1 -- 172.21.15.153:0/3623780320 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb28114c00 con 0x7ffb28078ce0 2024-03-11T22:53:20.304 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.716+0000 7ffb1e7fc700 1 -- 172.21.15.153:0/3623780320 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7ffb24019690 con 0x7ffb28078ce0 2024-03-11T22:53:20.305 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.716+0000 7ffb1e7fc700 1 -- 172.21.15.153:0/3623780320 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7ffb24024900 con 0x7ffb28078ce0 2024-03-11T22:53:20.305 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.716+0000 7ffb2ffb2700 1 -- 172.21.15.153:0/3623780320 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb2805f5c0 con 0x7ffb28078ce0 2024-03-11T22:53:20.305 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.718+0000 7ffb1e7fc700 1 -- 172.21.15.153:0/3623780320 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7ffb24029060 con 0x7ffb28078ce0 2024-03-11T22:53:20.305 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.783+0000 7ffb2ffb2700 1 -- 172.21.15.153:0/3623780320 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7ffb28108a40 con 0x7ffb28078ce0 2024-03-11T22:53:20.305 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.788+0000 7ffb1e7fc700 1 -- 172.21.15.153:0/3623780320 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7ffb2401a5f0 con 0x7ffb28078ce0 2024-03-11T22:53:20.305 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.789+0000 7ffb2ffb2700 1 -- 172.21.15.153:0/3623780320 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb28078ce0 msgr2=0x7ffb28113820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:20.305 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.789+0000 7ffb2ffb2700 1 --2- 172.21.15.153:0/3623780320 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb28078ce0 0x7ffb28113820 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ffb2400b440 tx=0x7ffb24009250 comp rx=0 tx=0).stop 2024-03-11T22:53:20.305 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.789+0000 7ffb2ffb2700 1 -- 172.21.15.153:0/3623780320 shutdown_connections 2024-03-11T22:53:20.305 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.789+0000 7ffb2ffb2700 1 --2- 172.21.15.153:0/3623780320 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffb28078ce0 0x7ffb28113820 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:20.305 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.789+0000 7ffb2ffb2700 1 -- 172.21.15.153:0/3623780320 >> 172.21.15.153:0/3623780320 conn(0x7ffb280fded0 msgr2=0x7ffb280ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:53:20.305 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.789+0000 7ffb2ffb2700 1 -- 172.21.15.153:0/3623780320 shutdown_connections 2024-03-11T22:53:20.305 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:19.789+0000 7ffb2ffb2700 1 -- 172.21.15.153:0/3623780320 wait complete. 2024-03-11T22:53:20.305 INFO:teuthology.orchestra.run.smithi153.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-11T22:53:20.305 INFO:teuthology.orchestra.run.smithi153.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:53:20.306 INFO:teuthology.orchestra.run.smithi153.stdout:Creating mgr... 2024-03-11T22:53:20.306 INFO:teuthology.orchestra.run.smithi153.stdout:Verifying port 9283 ... 2024-03-11T22:53:20.493 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-fa913b96-dff9-11ee-95c8-87774f69a715@mgr.smithi153.nyocep 2024-03-11T22:53:20.493 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Failed to reset failed state of unit ceph-fa913b96-dff9-11ee-95c8-87774f69a715@mgr.smithi153.nyocep.service: Unit ceph-fa913b96-dff9-11ee-95c8-87774f69a715@mgr.smithi153.nyocep.service not loaded. 2024-03-11T22:53:20.651 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-fa913b96-dff9-11ee-95c8-87774f69a715.target.wants/ceph-fa913b96-dff9-11ee-95c8-87774f69a715@mgr.smithi153.nyocep.service → /etc/systemd/system/ceph-fa913b96-dff9-11ee-95c8-87774f69a715@.service. 2024-03-11T22:53:21.074 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:20 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/3623780320' entity='client.admin' 2024-03-11T22:53:21.187 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T22:53:21.187 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout disabled 2024-03-11T22:53:21.195 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T22:53:21.195 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout inactive 2024-03-11T22:53:21.195 INFO:teuthology.orchestra.run.smithi153.stdout:firewalld.service is not enabled 2024-03-11T22:53:21.195 INFO:teuthology.orchestra.run.smithi153.stdout:Not possible to enable service . firewalld.service is not available 2024-03-11T22:53:21.204 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T22:53:21.204 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout disabled 2024-03-11T22:53:21.211 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T22:53:21.211 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout inactive 2024-03-11T22:53:21.212 INFO:teuthology.orchestra.run.smithi153.stdout:firewalld.service is not enabled 2024-03-11T22:53:21.212 INFO:teuthology.orchestra.run.smithi153.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2024-03-11T22:53:21.212 INFO:teuthology.orchestra.run.smithi153.stdout:Waiting for mgr to start... 2024-03-11T22:53:21.212 INFO:teuthology.orchestra.run.smithi153.stdout:Waiting for mgr... 2024-03-11T22:53:22.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:21 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/1566041999' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T22:53:22.250 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-03-11T22:53:22.253 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:53:22.253 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "fsid": "fa913b96-dff9-11ee-95c8-87774f69a715", 2024-03-11T22:53:22.253 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T22:53:22.253 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T22:53:22.253 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T22:53:22.253 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T22:53:22.253 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:22.253 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T22:53:22.253 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T22:53:22.253 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 0 2024-03-11T22:53:22.253 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:53:22.254 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T22:53:22.254 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "smithi153" 2024-03-11T22:53:22.254 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:53:22.254 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-11T22:53:22.254 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T22:53:22.254 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:53:22.254 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-03-11T22:53:22.254 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T22:53:22.254 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:22.254 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T22:53:22.254 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:53:22.254 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T22:53:22.254 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T22:53:22.262 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T22:53:22.262 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T22:53:22.262 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T22:53:22.262 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T22:53:22.262 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:22.262 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T22:53:22.262 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T22:53:22.262 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T22:53:22.262 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T22:53:22.263 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T22:53:22.263 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T22:53:22.263 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T22:53:22.263 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T22:53:22.263 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T22:53:22.263 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:22.263 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T22:53:22.263 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:53:22.263 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T22:53:22.263 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T22:53:22.263 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:22.263 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T22:53:22.263 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T22:53:22.263 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T22:53:22.264 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T22:53:22.264 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T22:53:22.264 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T22:53:22.264 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T22:53:22.264 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:53:22.264 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:53:22.264 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:22.264 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T22:53:22.264 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:53:22.264 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T22:53:14.269639+0000", 2024-03-11T22:53:22.264 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:53:22.264 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:22.264 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T22:53:22.264 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:53:22.264 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.586+0000 7f170e912700 1 Processor -- start 2024-03-11T22:53:22.265 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.586+0000 7f170e912700 1 -- start start 2024-03-11T22:53:22.265 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.587+0000 7f170e912700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17080714c0 0x7f1708105340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:22.265 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.587+0000 7f170e912700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1708105880 con 0x7f17080714c0 2024-03-11T22:53:22.265 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.587+0000 7f1707fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17080714c0 0x7f1708105340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:22.265 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.587+0000 7f1707fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17080714c0 0x7f1708105340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33850/0 (socket says 172.21.15.153:33850) 2024-03-11T22:53:22.265 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.587+0000 7f1707fff700 1 -- 172.21.15.153:0/3122950064 learned_addr learned my addr 172.21.15.153:0/3122950064 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:53:22.265 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.587+0000 7f1707fff700 1 -- 172.21.15.153:0/3122950064 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17081059c0 con 0x7f17080714c0 2024-03-11T22:53:22.265 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.588+0000 7f1707fff700 1 --2- 172.21.15.153:0/3122950064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17080714c0 0x7f1708105340 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f16f80098e0 tx=0x7f16f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fc2023f753c8dad server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:22.265 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.588+0000 7f1706ffd700 1 -- 172.21.15.153:0/3122950064 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16f800b860 con 0x7f17080714c0 2024-03-11T22:53:22.265 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.588+0000 7f1706ffd700 1 -- 172.21.15.153:0/3122950064 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f16f800be50 con 0x7f17080714c0 2024-03-11T22:53:22.265 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.588+0000 7f1706ffd700 1 -- 172.21.15.153:0/3122950064 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16f8010380 con 0x7f17080714c0 2024-03-11T22:53:22.265 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.588+0000 7f170e912700 1 -- 172.21.15.153:0/3122950064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17080714c0 msgr2=0x7f1708105340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:22.265 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.588+0000 7f170e912700 1 --2- 172.21.15.153:0/3122950064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17080714c0 0x7f1708105340 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f16f80098e0 tx=0x7f16f8009250 comp rx=0 tx=0).stop 2024-03-11T22:53:22.265 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.588+0000 7f170e912700 1 -- 172.21.15.153:0/3122950064 shutdown_connections 2024-03-11T22:53:22.265 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.588+0000 7f170e912700 1 --2- 172.21.15.153:0/3122950064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17080714c0 0x7f1708105340 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:22.266 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.588+0000 7f170e912700 1 -- 172.21.15.153:0/3122950064 >> 172.21.15.153:0/3122950064 conn(0x7f17080fcdd0 msgr2=0x7f17080ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:53:22.266 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.588+0000 7f170e912700 1 -- 172.21.15.153:0/3122950064 shutdown_connections 2024-03-11T22:53:22.266 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.588+0000 7f170e912700 1 -- 172.21.15.153:0/3122950064 wait complete. 2024-03-11T22:53:22.266 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.589+0000 7f170e912700 1 Processor -- start 2024-03-11T22:53:22.266 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.589+0000 7f170e912700 1 -- start start 2024-03-11T22:53:22.266 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.589+0000 7f170e912700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17080714c0 0x7f1708111580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:22.266 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.589+0000 7f170e912700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1708111ac0 con 0x7f17080714c0 2024-03-11T22:53:22.266 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.590+0000 7f1707fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17080714c0 0x7f1708111580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:22.266 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.590+0000 7f1707fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17080714c0 0x7f1708111580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33852/0 (socket says 172.21.15.153:33852) 2024-03-11T22:53:22.266 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.590+0000 7f1707fff700 1 -- 172.21.15.153:0/1566041999 learned_addr learned my addr 172.21.15.153:0/1566041999 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:53:22.266 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.590+0000 7f1707fff700 1 -- 172.21.15.153:0/1566041999 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f80094f0 con 0x7f17080714c0 2024-03-11T22:53:22.266 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.590+0000 7f1707fff700 1 --2- 172.21.15.153:0/1566041999 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17080714c0 0x7f1708111580 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f16f8003050 tx=0x7f16f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:22.266 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.590+0000 7f1704ff9700 1 -- 172.21.15.153:0/1566041999 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16f8010760 con 0x7f17080714c0 2024-03-11T22:53:22.266 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.590+0000 7f1704ff9700 1 -- 172.21.15.153:0/1566041999 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f16f8010d50 con 0x7f17080714c0 2024-03-11T22:53:22.267 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.590+0000 7f170e912700 1 -- 172.21.15.153:0/1566041999 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1708111cc0 con 0x7f17080714c0 2024-03-11T22:53:22.267 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.590+0000 7f1704ff9700 1 -- 172.21.15.153:0/1566041999 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16f801ae20 con 0x7f17080714c0 2024-03-11T22:53:22.267 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.590+0000 7f170e912700 1 -- 172.21.15.153:0/1566041999 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1708112960 con 0x7f17080714c0 2024-03-11T22:53:22.267 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.590+0000 7f1704ff9700 1 -- 172.21.15.153:0/1566041999 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f16f8024670 con 0x7f17080714c0 2024-03-11T22:53:22.267 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.591+0000 7f1704ff9700 1 -- 172.21.15.153:0/1566041999 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f16f8024c70 con 0x7f17080714c0 2024-03-11T22:53:22.267 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.591+0000 7f170e912700 1 -- 172.21.15.153:0/1566041999 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f170805f5c0 con 0x7f17080714c0 2024-03-11T22:53:22.267 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.594+0000 7f1704ff9700 1 -- 172.21.15.153:0/1566041999 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f16f8029070 con 0x7f17080714c0 2024-03-11T22:53:22.267 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.665+0000 7f170e912700 1 -- 172.21.15.153:0/1566041999 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f1708106800 con 0x7f17080714c0 2024-03-11T22:53:22.267 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.666+0000 7f1704ff9700 1 -- 172.21.15.153:0/1566041999 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f16f8010ec0 con 0x7f17080714c0 2024-03-11T22:53:22.267 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.667+0000 7f170e912700 1 -- 172.21.15.153:0/1566041999 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17080714c0 msgr2=0x7f1708111580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:22.267 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.667+0000 7f170e912700 1 --2- 172.21.15.153:0/1566041999 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17080714c0 0x7f1708111580 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f16f8003050 tx=0x7f16f8009250 comp rx=0 tx=0).stop 2024-03-11T22:53:22.267 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.667+0000 7f170e912700 1 -- 172.21.15.153:0/1566041999 shutdown_connections 2024-03-11T22:53:22.267 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.667+0000 7f170e912700 1 --2- 172.21.15.153:0/1566041999 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f17080714c0 0x7f1708111580 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:22.267 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.667+0000 7f170e912700 1 -- 172.21.15.153:0/1566041999 >> 172.21.15.153:0/1566041999 conn(0x7f17080fcdd0 msgr2=0x7f17080fe540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:53:22.268 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.667+0000 7f170e912700 1 -- 172.21.15.153:0/1566041999 shutdown_connections 2024-03-11T22:53:22.268 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:21.667+0000 7f170e912700 1 -- 172.21.15.153:0/1566041999 wait complete. 2024-03-11T22:53:22.268 INFO:teuthology.orchestra.run.smithi153.stdout:mgr not available, waiting (1/15)... 2024-03-11T22:53:25.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:25 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/1082788926' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T22:53:25.376 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-03-11T22:53:25.376 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:53:25.376 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "fsid": "fa913b96-dff9-11ee-95c8-87774f69a715", 2024-03-11T22:53:25.376 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T22:53:25.376 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T22:53:25.376 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T22:53:25.377 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T22:53:25.377 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:25.377 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T22:53:25.377 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T22:53:25.377 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 0 2024-03-11T22:53:25.377 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:53:25.377 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T22:53:25.377 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "smithi153" 2024-03-11T22:53:25.377 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:53:25.377 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-11T22:53:25.377 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T22:53:25.377 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:53:25.377 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-03-11T22:53:25.377 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T22:53:25.377 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:25.378 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T22:53:25.378 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:53:25.378 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T22:53:25.378 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T22:53:25.386 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T22:53:25.386 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T22:53:25.386 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T22:53:25.386 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T22:53:25.386 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:25.386 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T22:53:25.386 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T22:53:25.386 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T22:53:25.386 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T22:53:25.386 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T22:53:25.386 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T22:53:25.387 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T22:53:25.387 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T22:53:25.387 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T22:53:25.387 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:25.387 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T22:53:25.387 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:53:25.387 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T22:53:25.387 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T22:53:25.387 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:25.387 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T22:53:25.387 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T22:53:25.387 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T22:53:25.387 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T22:53:25.388 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T22:53:25.388 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T22:53:25.388 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T22:53:25.388 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:53:25.388 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:53:25.388 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:25.388 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T22:53:25.388 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:53:25.388 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T22:53:14.269639+0000", 2024-03-11T22:53:25.388 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:53:25.388 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:25.388 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T22:53:25.388 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:53:25.388 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.667+0000 7f42cebe1700 1 Processor -- start 2024-03-11T22:53:25.388 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.667+0000 7f42cebe1700 1 -- start start 2024-03-11T22:53:25.388 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.667+0000 7f42cebe1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42c8101670 0x7f42c8101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:25.389 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.667+0000 7f42cebe1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c8101fc0 con 0x7f42c8101670 2024-03-11T22:53:25.389 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.667+0000 7f42cc97d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42c8101670 0x7f42c8101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:25.389 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.667+0000 7f42cc97d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42c8101670 0x7f42c8101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33864/0 (socket says 172.21.15.153:33864) 2024-03-11T22:53:25.389 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.667+0000 7f42cc97d700 1 -- 172.21.15.153:0/2519983379 learned_addr learned my addr 172.21.15.153:0/2519983379 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:53:25.389 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.668+0000 7f42cc97d700 1 -- 172.21.15.153:0/2519983379 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c8102100 con 0x7f42c8101670 2024-03-11T22:53:25.389 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.668+0000 7f42cc97d700 1 --2- 172.21.15.153:0/2519983379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42c8101670 0x7f42c8101a80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f42bc0098e0 tx=0x7f42bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e337d134be3411e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:25.389 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.668+0000 7f42c77fe700 1 -- 172.21.15.153:0/2519983379 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42bc00b860 con 0x7f42c8101670 2024-03-11T22:53:25.389 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.668+0000 7f42c77fe700 1 -- 172.21.15.153:0/2519983379 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f42bc00be50 con 0x7f42c8101670 2024-03-11T22:53:25.389 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.668+0000 7f42c77fe700 1 -- 172.21.15.153:0/2519983379 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42bc010380 con 0x7f42c8101670 2024-03-11T22:53:25.389 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.669+0000 7f42cebe1700 1 -- 172.21.15.153:0/2519983379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42c8101670 msgr2=0x7f42c8101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:25.389 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.669+0000 7f42cebe1700 1 --2- 172.21.15.153:0/2519983379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42c8101670 0x7f42c8101a80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f42bc0098e0 tx=0x7f42bc009250 comp rx=0 tx=0).stop 2024-03-11T22:53:25.389 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.669+0000 7f42cebe1700 1 -- 172.21.15.153:0/2519983379 shutdown_connections 2024-03-11T22:53:25.389 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.669+0000 7f42cebe1700 1 --2- 172.21.15.153:0/2519983379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42c8101670 0x7f42c8101a80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:25.389 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.669+0000 7f42cebe1700 1 -- 172.21.15.153:0/2519983379 >> 172.21.15.153:0/2519983379 conn(0x7f42c80fcdd0 msgr2=0x7f42c80ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:53:25.390 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.669+0000 7f42cebe1700 1 -- 172.21.15.153:0/2519983379 shutdown_connections 2024-03-11T22:53:25.390 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.669+0000 7f42cebe1700 1 -- 172.21.15.153:0/2519983379 wait complete. 2024-03-11T22:53:25.390 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.669+0000 7f42cebe1700 1 Processor -- start 2024-03-11T22:53:25.390 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.669+0000 7f42cebe1700 1 -- start start 2024-03-11T22:53:25.390 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.670+0000 7f42cebe1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42c8101670 0x7f42c8111540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:25.390 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.670+0000 7f42cebe1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42c8111a80 con 0x7f42c8101670 2024-03-11T22:53:25.390 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.670+0000 7f42cc97d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42c8101670 0x7f42c8111540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:25.390 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.670+0000 7f42cc97d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42c8101670 0x7f42c8111540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33876/0 (socket says 172.21.15.153:33876) 2024-03-11T22:53:25.390 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.670+0000 7f42cc97d700 1 -- 172.21.15.153:0/1082788926 learned_addr learned my addr 172.21.15.153:0/1082788926 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:53:25.390 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.670+0000 7f42cc97d700 1 -- 172.21.15.153:0/1082788926 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42bc0094f0 con 0x7f42c8101670 2024-03-11T22:53:25.390 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.670+0000 7f42cc97d700 1 --2- 172.21.15.153:0/1082788926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42c8101670 0x7f42c8111540 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f42bc00b440 tx=0x7f42bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:25.390 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.670+0000 7f42c57fa700 1 -- 172.21.15.153:0/1082788926 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42bc010760 con 0x7f42c8101670 2024-03-11T22:53:25.390 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.670+0000 7f42c57fa700 1 -- 172.21.15.153:0/1082788926 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f42bc010d50 con 0x7f42c8101670 2024-03-11T22:53:25.391 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.670+0000 7f42cebe1700 1 -- 172.21.15.153:0/1082788926 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42c8111c80 con 0x7f42c8101670 2024-03-11T22:53:25.391 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.670+0000 7f42c57fa700 1 -- 172.21.15.153:0/1082788926 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42bc01ae20 con 0x7f42c8101670 2024-03-11T22:53:25.391 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.670+0000 7f42cebe1700 1 -- 172.21.15.153:0/1082788926 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42c8112920 con 0x7f42c8101670 2024-03-11T22:53:25.391 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.671+0000 7f42c57fa700 1 -- 172.21.15.153:0/1082788926 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f42bc02e6c0 con 0x7f42c8101670 2024-03-11T22:53:25.391 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.671+0000 7f42c57fa700 1 -- 172.21.15.153:0/1082788926 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f42bc024950 con 0x7f42c8101670 2024-03-11T22:53:25.391 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.671+0000 7f42cebe1700 1 -- 172.21.15.153:0/1082788926 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42c805f5c0 con 0x7f42c8101670 2024-03-11T22:53:25.391 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.673+0000 7f42c57fa700 1 -- 172.21.15.153:0/1082788926 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f42bc029070 con 0x7f42c8101670 2024-03-11T22:53:25.391 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.744+0000 7f42cebe1700 1 -- 172.21.15.153:0/1082788926 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f42c81034e0 con 0x7f42c8101670 2024-03-11T22:53:25.391 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.745+0000 7f42c57fa700 1 -- 172.21.15.153:0/1082788926 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f42bc01a6a0 con 0x7f42c8101670 2024-03-11T22:53:25.391 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.746+0000 7f42cebe1700 1 -- 172.21.15.153:0/1082788926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42c8101670 msgr2=0x7f42c8111540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:25.391 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.746+0000 7f42cebe1700 1 --2- 172.21.15.153:0/1082788926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42c8101670 0x7f42c8111540 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f42bc00b440 tx=0x7f42bc009250 comp rx=0 tx=0).stop 2024-03-11T22:53:25.391 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.746+0000 7f42cebe1700 1 -- 172.21.15.153:0/1082788926 shutdown_connections 2024-03-11T22:53:25.391 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.746+0000 7f42cebe1700 1 --2- 172.21.15.153:0/1082788926 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42c8101670 0x7f42c8111540 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:25.392 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.746+0000 7f42cebe1700 1 -- 172.21.15.153:0/1082788926 >> 172.21.15.153:0/1082788926 conn(0x7f42c80fcdd0 msgr2=0x7f42c8076850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:53:25.392 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.746+0000 7f42cebe1700 1 -- 172.21.15.153:0/1082788926 shutdown_connections 2024-03-11T22:53:25.392 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:24.746+0000 7f42cebe1700 1 -- 172.21.15.153:0/1082788926 wait complete. 2024-03-11T22:53:25.392 INFO:teuthology.orchestra.run.smithi153.stdout:mgr not available, waiting (2/15)... 2024-03-11T22:53:28.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:27 smithi153 ceph-mon[28962]: Activating manager daemon smithi153.nyocep 2024-03-11T22:53:28.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:27 smithi153 ceph-mon[28962]: mgrmap e2: smithi153.nyocep(active, starting, since 0.00293141s) 2024-03-11T22:53:28.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:27 smithi153 ceph-mon[28962]: from='mgr.14100 172.21.15.153:0/3669562610' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:53:28.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:27 smithi153 ceph-mon[28962]: from='mgr.14100 172.21.15.153:0/3669562610' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:53:28.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:27 smithi153 ceph-mon[28962]: from='mgr.14100 172.21.15.153:0/3669562610' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:53:28.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:27 smithi153 ceph-mon[28962]: from='mgr.14100 172.21.15.153:0/3669562610' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-03-11T22:53:28.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:27 smithi153 ceph-mon[28962]: from='mgr.14100 172.21.15.153:0/3669562610' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mgr metadata", "who": "smithi153.nyocep", "id": "smithi153.nyocep"}]: dispatch 2024-03-11T22:53:28.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:27 smithi153 ceph-mon[28962]: Manager daemon smithi153.nyocep is now available 2024-03-11T22:53:28.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:27 smithi153 ceph-mon[28962]: from='mgr.14100 172.21.15.153:0/3669562610' entity='mgr.smithi153.nyocep' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.nyocep/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:53:28.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:27 smithi153 ceph-mon[28962]: from='mgr.14100 172.21.15.153:0/3669562610' entity='mgr.smithi153.nyocep' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.nyocep/trash_purge_schedule"}]: dispatch 2024-03-11T22:53:28.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:27 smithi153 ceph-mon[28962]: from='mgr.14100 172.21.15.153:0/3669562610' entity='mgr.smithi153.nyocep' 2024-03-11T22:53:28.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:27 smithi153 ceph-mon[28962]: from='mgr.14100 172.21.15.153:0/3669562610' entity='mgr.smithi153.nyocep' 2024-03-11T22:53:28.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:27 smithi153 ceph-mon[28962]: from='mgr.14100 172.21.15.153:0/3669562610' entity='mgr.smithi153.nyocep' 2024-03-11T22:53:29.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:29 smithi153 ceph-mon[28962]: mgrmap e3: smithi153.nyocep(active, since 1.00699s) 2024-03-11T22:53:31.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:31 smithi153 ceph-mon[28962]: mgrmap e4: smithi153.nyocep(active, since 3s) 2024-03-11T22:53:31.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:31 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/801797903' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T22:53:37.007 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-03-11T22:53:37.007 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:53:37.007 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "fsid": "fa913b96-dff9-11ee-95c8-87774f69a715", 2024-03-11T22:53:37.007 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T22:53:37.007 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T22:53:37.008 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T22:53:37.008 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T22:53:37.008 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:37.008 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T22:53:37.008 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T22:53:37.008 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 0 2024-03-11T22:53:37.008 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:53:37.008 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T22:53:37.008 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "smithi153" 2024-03-11T22:53:37.008 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:53:37.008 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2024-03-11T22:53:37.008 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T22:53:37.008 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:53:37.008 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-03-11T22:53:37.008 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T22:53:37.009 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:37.009 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T22:53:37.009 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:53:37.009 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T22:53:37.009 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T22:53:37.009 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T22:53:37.009 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T22:53:37.018 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T22:53:37.018 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T22:53:37.018 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:37.018 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T22:53:37.018 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T22:53:37.018 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T22:53:37.018 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T22:53:37.018 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T22:53:37.018 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T22:53:37.018 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T22:53:37.018 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T22:53:37.018 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T22:53:37.018 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:37.018 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T22:53:37.019 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:53:37.019 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T22:53:37.019 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T22:53:37.019 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:37.019 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T22:53:37.019 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T22:53:37.019 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T22:53:37.019 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T22:53:37.019 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T22:53:37.019 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T22:53:37.019 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T22:53:37.019 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:53:37.019 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:53:37.019 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:37.019 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T22:53:37.020 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:53:37.020 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T22:53:14.269639+0000", 2024-03-11T22:53:37.020 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:53:37.020 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:37.020 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T22:53:37.020 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:53:37.020 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.653+0000 7febac3c0700 1 Processor -- start 2024-03-11T22:53:37.020 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.653+0000 7febac3c0700 1 -- start start 2024-03-11T22:53:37.020 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.653+0000 7febac3c0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feba4076c60 0x7feba4105340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:37.020 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.653+0000 7febac3c0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba4105880 con 0x7feba4076c60 2024-03-11T22:53:37.020 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.653+0000 7febaa15c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feba4076c60 0x7feba4105340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:37.020 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.653+0000 7febaa15c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feba4076c60 0x7feba4105340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33938/0 (socket says 172.21.15.153:33938) 2024-03-11T22:53:37.020 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.653+0000 7febaa15c700 1 -- 172.21.15.153:0/20010325 learned_addr learned my addr 172.21.15.153:0/20010325 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:53:37.020 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.654+0000 7febaa15c700 1 -- 172.21.15.153:0/20010325 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba41059c0 con 0x7feba4076c60 2024-03-11T22:53:37.020 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.654+0000 7febaa15c700 1 --2- 172.21.15.153:0/20010325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feba4076c60 0x7feba4105340 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7feba00098e0 tx=0x7feba0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6a74b0cc7c5cf3f server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:37.020 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.654+0000 7feba915a700 1 -- 172.21.15.153:0/20010325 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feba000b860 con 0x7feba4076c60 2024-03-11T22:53:37.021 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.654+0000 7feba915a700 1 -- 172.21.15.153:0/20010325 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7feba000be50 con 0x7feba4076c60 2024-03-11T22:53:37.021 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.654+0000 7feba915a700 1 -- 172.21.15.153:0/20010325 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feba0010380 con 0x7feba4076c60 2024-03-11T22:53:37.021 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.655+0000 7febac3c0700 1 -- 172.21.15.153:0/20010325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feba4076c60 msgr2=0x7feba4105340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:37.021 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.655+0000 7febac3c0700 1 --2- 172.21.15.153:0/20010325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feba4076c60 0x7feba4105340 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7feba00098e0 tx=0x7feba0009250 comp rx=0 tx=0).stop 2024-03-11T22:53:37.021 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.655+0000 7febac3c0700 1 -- 172.21.15.153:0/20010325 shutdown_connections 2024-03-11T22:53:37.021 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.655+0000 7febac3c0700 1 --2- 172.21.15.153:0/20010325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feba4076c60 0x7feba4105340 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:37.021 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.655+0000 7febac3c0700 1 -- 172.21.15.153:0/20010325 >> 172.21.15.153:0/20010325 conn(0x7feba40fcdd0 msgr2=0x7feba40ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:53:37.021 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.655+0000 7febac3c0700 1 -- 172.21.15.153:0/20010325 shutdown_connections 2024-03-11T22:53:37.021 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.655+0000 7febac3c0700 1 -- 172.21.15.153:0/20010325 wait complete. 2024-03-11T22:53:37.021 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.655+0000 7febac3c0700 1 Processor -- start 2024-03-11T22:53:37.021 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.655+0000 7febac3c0700 1 -- start start 2024-03-11T22:53:37.021 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.655+0000 7febac3c0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feba4076c60 0x7feba4111820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:37.021 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.655+0000 7febac3c0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feba4111d60 con 0x7feba4076c60 2024-03-11T22:53:37.021 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.656+0000 7febaa15c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feba4076c60 0x7feba4111820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:37.021 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.656+0000 7febaa15c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feba4076c60 0x7feba4111820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33944/0 (socket says 172.21.15.153:33944) 2024-03-11T22:53:37.022 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.656+0000 7febaa15c700 1 -- 172.21.15.153:0/801797903 learned_addr learned my addr 172.21.15.153:0/801797903 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:53:37.022 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.656+0000 7febaa15c700 1 -- 172.21.15.153:0/801797903 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba00094f0 con 0x7feba4076c60 2024-03-11T22:53:37.022 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.656+0000 7febaa15c700 1 --2- 172.21.15.153:0/801797903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feba4076c60 0x7feba4111820 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7feba000b440 tx=0x7feba0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:37.022 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.656+0000 7feb9affd700 1 -- 172.21.15.153:0/801797903 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feba00107c0 con 0x7feba4076c60 2024-03-11T22:53:37.022 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.656+0000 7feb9affd700 1 -- 172.21.15.153:0/801797903 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7feba0010db0 con 0x7feba4076c60 2024-03-11T22:53:37.022 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.656+0000 7febac3c0700 1 -- 172.21.15.153:0/801797903 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feba4111f60 con 0x7feba4076c60 2024-03-11T22:53:37.022 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.656+0000 7feb9affd700 1 -- 172.21.15.153:0/801797903 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feba001ad30 con 0x7feba4076c60 2024-03-11T22:53:37.022 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.656+0000 7febac3c0700 1 -- 172.21.15.153:0/801797903 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feba4112c00 con 0x7feba4076c60 2024-03-11T22:53:37.022 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.657+0000 7feb9affd700 1 -- 172.21.15.153:0/801797903 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7feba002e440 con 0x7feba4076c60 2024-03-11T22:53:37.022 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.657+0000 7febac3c0700 1 -- 172.21.15.153:0/801797903 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feba405f5c0 con 0x7feba4076c60 2024-03-11T22:53:37.022 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.657+0000 7feb9affd700 1 --2- 172.21.15.153:0/801797903 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7feb90036740 0x7feb90038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:37.022 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.657+0000 7feb9affd700 1 -- 172.21.15.153:0/801797903 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7feba004adf0 con 0x7feba4076c60 2024-03-11T22:53:37.022 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.658+0000 7feba995b700 1 --2- 172.21.15.153:0/801797903 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7feb90036740 0x7feb90038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:37.022 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.658+0000 7feba995b700 1 --2- 172.21.15.153:0/801797903 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7feb90036740 0x7feb90038bf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7feb94009990 tx=0x7feb94006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:37.023 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.662+0000 7feb9affd700 1 -- 172.21.15.153:0/801797903 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7feba002c980 con 0x7feba4076c60 2024-03-11T22:53:37.023 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.927+0000 7febac3c0700 1 -- 172.21.15.153:0/801797903 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7feba404bed0 con 0x7feba4076c60 2024-03-11T22:53:37.023 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.928+0000 7feb9affd700 1 -- 172.21.15.153:0/801797903 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7feba002c390 con 0x7feba4076c60 2024-03-11T22:53:37.023 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.931+0000 7febac3c0700 1 -- 172.21.15.153:0/801797903 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7feb90036740 msgr2=0x7feb90038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:37.023 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.931+0000 7febac3c0700 1 --2- 172.21.15.153:0/801797903 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7feb90036740 0x7feb90038bf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7feb94009990 tx=0x7feb94006e40 comp rx=0 tx=0).stop 2024-03-11T22:53:37.023 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.931+0000 7febac3c0700 1 -- 172.21.15.153:0/801797903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feba4076c60 msgr2=0x7feba4111820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:37.023 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.931+0000 7febac3c0700 1 --2- 172.21.15.153:0/801797903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feba4076c60 0x7feba4111820 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7feba000b440 tx=0x7feba0009250 comp rx=0 tx=0).stop 2024-03-11T22:53:37.023 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.931+0000 7febac3c0700 1 -- 172.21.15.153:0/801797903 shutdown_connections 2024-03-11T22:53:37.023 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.931+0000 7febac3c0700 1 --2- 172.21.15.153:0/801797903 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7feb90036740 0x7feb90038bf0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:37.023 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.931+0000 7febac3c0700 1 --2- 172.21.15.153:0/801797903 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feba4076c60 0x7feba4111820 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:37.023 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.931+0000 7febac3c0700 1 -- 172.21.15.153:0/801797903 >> 172.21.15.153:0/801797903 conn(0x7feba40fcdd0 msgr2=0x7feba40ff0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:53:37.023 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.931+0000 7febac3c0700 1 -- 172.21.15.153:0/801797903 shutdown_connections 2024-03-11T22:53:37.023 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:30.931+0000 7febac3c0700 1 -- 172.21.15.153:0/801797903 wait complete. 2024-03-11T22:53:37.023 INFO:teuthology.orchestra.run.smithi153.stdout:mgr is available 2024-03-11T22:53:38.428 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:38 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/2093250712' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-11T22:53:38.681 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-03-11T22:53:38.689 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout [global] 2024-03-11T22:53:38.690 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout fsid = fa913b96-dff9-11ee-95c8-87774f69a715 2024-03-11T22:53:38.690 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-11T22:53:38.690 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-11T22:53:38.690 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-11T22:53:38.690 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-11T22:53:38.690 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-03-11T22:53:38.690 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-11T22:53:38.690 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-11T22:53:38.690 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-03-11T22:53:38.690 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout [osd] 2024-03-11T22:53:38.690 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-11T22:53:38.690 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-11T22:53:38.691 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-11T22:53:38.691 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-11T22:53:38.691 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-11T22:53:38.691 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.868+0000 7f2bfbecf700 1 Processor -- start 2024-03-11T22:53:38.691 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.868+0000 7f2bfbecf700 1 -- start start 2024-03-11T22:53:38.691 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.868+0000 7f2bfbecf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bf4076c40 0x7f2bf4105360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:38.691 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.868+0000 7f2bfbecf700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf41058a0 con 0x7f2bf4076c40 2024-03-11T22:53:38.691 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.868+0000 7f2bf9c6b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bf4076c40 0x7f2bf4105360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:38.691 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.869+0000 7f2bf9c6b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bf4076c40 0x7f2bf4105360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54868/0 (socket says 172.21.15.153:54868) 2024-03-11T22:53:38.691 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.869+0000 7f2bf9c6b700 1 -- 172.21.15.153:0/2008563673 learned_addr learned my addr 172.21.15.153:0/2008563673 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:53:38.691 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.869+0000 7f2bf9c6b700 1 -- 172.21.15.153:0/2008563673 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf41059e0 con 0x7f2bf4076c40 2024-03-11T22:53:38.691 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.869+0000 7f2bf9c6b700 1 --2- 172.21.15.153:0/2008563673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bf4076c40 0x7f2bf4105360 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2bf00098e0 tx=0x7f2bf0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76cbdd57a933c58a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:38.691 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.869+0000 7f2bf8c69700 1 -- 172.21.15.153:0/2008563673 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bf000b860 con 0x7f2bf4076c40 2024-03-11T22:53:38.692 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.869+0000 7f2bf8c69700 1 -- 172.21.15.153:0/2008563673 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f2bf000be50 con 0x7f2bf4076c40 2024-03-11T22:53:38.692 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.869+0000 7f2bf8c69700 1 -- 172.21.15.153:0/2008563673 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bf0010380 con 0x7f2bf4076c40 2024-03-11T22:53:38.692 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.870+0000 7f2bfbecf700 1 -- 172.21.15.153:0/2008563673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bf4076c40 msgr2=0x7f2bf4105360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:38.692 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.870+0000 7f2bfbecf700 1 --2- 172.21.15.153:0/2008563673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bf4076c40 0x7f2bf4105360 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2bf00098e0 tx=0x7f2bf0009250 comp rx=0 tx=0).stop 2024-03-11T22:53:38.692 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.870+0000 7f2bfbecf700 1 -- 172.21.15.153:0/2008563673 shutdown_connections 2024-03-11T22:53:38.692 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.870+0000 7f2bfbecf700 1 --2- 172.21.15.153:0/2008563673 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bf4076c40 0x7f2bf4105360 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:38.692 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.870+0000 7f2bfbecf700 1 -- 172.21.15.153:0/2008563673 >> 172.21.15.153:0/2008563673 conn(0x7f2bf40fcdd0 msgr2=0x7f2bf40ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:53:38.692 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.870+0000 7f2bfbecf700 1 -- 172.21.15.153:0/2008563673 shutdown_connections 2024-03-11T22:53:38.692 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.870+0000 7f2bfbecf700 1 -- 172.21.15.153:0/2008563673 wait complete. 2024-03-11T22:53:38.692 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.870+0000 7f2bfbecf700 1 Processor -- start 2024-03-11T22:53:38.692 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.870+0000 7f2bfbecf700 1 -- start start 2024-03-11T22:53:38.692 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.871+0000 7f2bfbecf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bf4076c40 0x7f2bf4111820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:38.692 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.871+0000 7f2bfbecf700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf4111d60 con 0x7f2bf4076c40 2024-03-11T22:53:38.692 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.871+0000 7f2bf9c6b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bf4076c40 0x7f2bf4111820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:38.692 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.871+0000 7f2bf9c6b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bf4076c40 0x7f2bf4111820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54880/0 (socket says 172.21.15.153:54880) 2024-03-11T22:53:38.693 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.871+0000 7f2bf9c6b700 1 -- 172.21.15.153:0/2093250712 learned_addr learned my addr 172.21.15.153:0/2093250712 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:53:38.693 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.871+0000 7f2bf9c6b700 1 -- 172.21.15.153:0/2093250712 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf00094f0 con 0x7f2bf4076c40 2024-03-11T22:53:38.693 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.871+0000 7f2bf9c6b700 1 --2- 172.21.15.153:0/2093250712 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bf4076c40 0x7f2bf4111820 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2bf0006d40 tx=0x7f2bf0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:38.693 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.871+0000 7f2bea7fc700 1 -- 172.21.15.153:0/2093250712 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bf0010760 con 0x7f2bf4076c40 2024-03-11T22:53:38.693 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.871+0000 7f2bea7fc700 1 -- 172.21.15.153:0/2093250712 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f2bf0010d50 con 0x7f2bf4076c40 2024-03-11T22:53:38.693 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.871+0000 7f2bea7fc700 1 -- 172.21.15.153:0/2093250712 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bf0024d20 con 0x7f2bf4076c40 2024-03-11T22:53:38.693 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.871+0000 7f2bfbecf700 1 -- 172.21.15.153:0/2093250712 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bf4111f60 con 0x7f2bf4076c40 2024-03-11T22:53:38.693 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.871+0000 7f2bfbecf700 1 -- 172.21.15.153:0/2093250712 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bf4112c00 con 0x7f2bf4076c40 2024-03-11T22:53:38.693 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.872+0000 7f2bea7fc700 1 -- 172.21.15.153:0/2093250712 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7f2bf00108c0 con 0x7f2bf4076c40 2024-03-11T22:53:38.693 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.872+0000 7f2bfbecf700 1 -- 172.21.15.153:0/2093250712 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bf405f5c0 con 0x7f2bf4076c40 2024-03-11T22:53:38.693 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.872+0000 7f2bea7fc700 1 --2- 172.21.15.153:0/2093250712 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7f2be0036790 0x7f2be0038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:38.693 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.872+0000 7f2bea7fc700 1 -- 172.21.15.153:0/2093250712 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f2bf004a710 con 0x7f2bf4076c40 2024-03-11T22:53:38.693 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.873+0000 7f2bf946a700 1 --2- 172.21.15.153:0/2093250712 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7f2be0036790 0x7f2be0038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:38.694 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.873+0000 7f2bf946a700 1 --2- 172.21.15.153:0/2093250712 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7f2be0036790 0x7f2be0038c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2be4009990 tx=0x7f2be4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:38.694 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:37.877+0000 7f2bea7fc700 1 -- 172.21.15.153:0/2093250712 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f2bf002c990 con 0x7f2bf4076c40 2024-03-11T22:53:38.694 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:38.059+0000 7f2bfbecf700 1 -- 172.21.15.153:0/2093250712 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f2bf404bed0 con 0x7f2bf4076c40 2024-03-11T22:53:38.694 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:38.060+0000 7f2bea7fc700 1 -- 172.21.15.153:0/2093250712 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f2bf000ee60 con 0x7f2bf4076c40 2024-03-11T22:53:38.694 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:38.062+0000 7f2bfbecf700 1 -- 172.21.15.153:0/2093250712 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7f2be0036790 msgr2=0x7f2be0038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:38.694 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:38.062+0000 7f2bfbecf700 1 --2- 172.21.15.153:0/2093250712 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7f2be0036790 0x7f2be0038c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2be4009990 tx=0x7f2be4006e40 comp rx=0 tx=0).stop 2024-03-11T22:53:38.694 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:38.062+0000 7f2bfbecf700 1 -- 172.21.15.153:0/2093250712 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bf4076c40 msgr2=0x7f2bf4111820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:38.694 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:38.062+0000 7f2bfbecf700 1 --2- 172.21.15.153:0/2093250712 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bf4076c40 0x7f2bf4111820 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2bf0006d40 tx=0x7f2bf0009250 comp rx=0 tx=0).stop 2024-03-11T22:53:38.694 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:38.062+0000 7f2bfbecf700 1 -- 172.21.15.153:0/2093250712 shutdown_connections 2024-03-11T22:53:38.694 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:38.062+0000 7f2bfbecf700 1 --2- 172.21.15.153:0/2093250712 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7f2be0036790 0x7f2be0038c40 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:38.694 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:38.062+0000 7f2bfbecf700 1 --2- 172.21.15.153:0/2093250712 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bf4076c40 0x7f2bf4111820 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:38.694 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:38.062+0000 7f2bfbecf700 1 -- 172.21.15.153:0/2093250712 >> 172.21.15.153:0/2093250712 conn(0x7f2bf40fcdd0 msgr2=0x7f2bf40ff0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:53:38.694 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:38.062+0000 7f2bfbecf700 1 -- 172.21.15.153:0/2093250712 shutdown_connections 2024-03-11T22:53:38.694 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:38.062+0000 7f2bfbecf700 1 -- 172.21.15.153:0/2093250712 wait complete. 2024-03-11T22:53:38.695 INFO:teuthology.orchestra.run.smithi153.stdout:Enabling cephadm module... 2024-03-11T22:53:40.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:40 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/3404712345' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-11T22:53:40.776 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.134+0000 7fda62428700 1 Processor -- start 2024-03-11T22:53:40.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.134+0000 7fda62428700 1 -- start start 2024-03-11T22:53:40.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.134+0000 7fda62428700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda5c101670 0x7fda5c101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:40.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.135+0000 7fda62428700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5c101fc0 con 0x7fda5c101670 2024-03-11T22:53:40.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.135+0000 7fda5bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda5c101670 0x7fda5c101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:40.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.135+0000 7fda5bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda5c101670 0x7fda5c101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54882/0 (socket says 172.21.15.153:54882) 2024-03-11T22:53:40.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.135+0000 7fda5bfff700 1 -- 172.21.15.153:0/2482337060 learned_addr learned my addr 172.21.15.153:0/2482337060 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:53:40.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.135+0000 7fda5bfff700 1 -- 172.21.15.153:0/2482337060 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda5c102100 con 0x7fda5c101670 2024-03-11T22:53:40.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.135+0000 7fda5bfff700 1 --2- 172.21.15.153:0/2482337060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda5c101670 0x7fda5c101a80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fda4c00bb80 tx=0x7fda4c00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c184e2c4f0925e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:40.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.135+0000 7fda5affd700 1 -- 172.21.15.153:0/2482337060 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda4c00dd00 con 0x7fda5c101670 2024-03-11T22:53:40.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.135+0000 7fda5affd700 1 -- 172.21.15.153:0/2482337060 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fda4c00de60 con 0x7fda5c101670 2024-03-11T22:53:40.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.135+0000 7fda5affd700 1 -- 172.21.15.153:0/2482337060 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda4c0188a0 con 0x7fda5c101670 2024-03-11T22:53:40.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.136+0000 7fda62428700 1 -- 172.21.15.153:0/2482337060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda5c101670 msgr2=0x7fda5c101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:40.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.136+0000 7fda62428700 1 --2- 172.21.15.153:0/2482337060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda5c101670 0x7fda5c101a80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fda4c00bb80 tx=0x7fda4c00b4f0 comp rx=0 tx=0).stop 2024-03-11T22:53:40.777 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.136+0000 7fda62428700 1 -- 172.21.15.153:0/2482337060 shutdown_connections 2024-03-11T22:53:40.778 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.136+0000 7fda62428700 1 --2- 172.21.15.153:0/2482337060 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda5c101670 0x7fda5c101a80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:40.778 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.136+0000 7fda62428700 1 -- 172.21.15.153:0/2482337060 >> 172.21.15.153:0/2482337060 conn(0x7fda5c0fcdd0 msgr2=0x7fda5c0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:53:40.778 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.136+0000 7fda62428700 1 -- 172.21.15.153:0/2482337060 shutdown_connections 2024-03-11T22:53:40.778 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.136+0000 7fda62428700 1 -- 172.21.15.153:0/2482337060 wait complete. 2024-03-11T22:53:40.778 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.136+0000 7fda62428700 1 Processor -- start 2024-03-11T22:53:40.778 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.137+0000 7fda62428700 1 -- start start 2024-03-11T22:53:40.778 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.137+0000 7fda62428700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda5c101670 0x7fda5c1136e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:40.778 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.137+0000 7fda62428700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5c113c20 con 0x7fda5c101670 2024-03-11T22:53:40.778 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.137+0000 7fda5bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda5c101670 0x7fda5c1136e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:40.778 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.137+0000 7fda5bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda5c101670 0x7fda5c1136e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54890/0 (socket says 172.21.15.153:54890) 2024-03-11T22:53:40.778 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.137+0000 7fda5bfff700 1 -- 172.21.15.153:0/3404712345 learned_addr learned my addr 172.21.15.153:0/3404712345 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:53:40.778 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.137+0000 7fda5bfff700 1 -- 172.21.15.153:0/3404712345 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda4c00b790 con 0x7fda5c101670 2024-03-11T22:53:40.778 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.137+0000 7fda5bfff700 1 --2- 172.21.15.153:0/3404712345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda5c101670 0x7fda5c1136e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fda4c006b00 tx=0x7fda4c00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:40.778 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.137+0000 7fda58ff9700 1 -- 172.21.15.153:0/3404712345 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda4c018e60 con 0x7fda5c101670 2024-03-11T22:53:40.779 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.137+0000 7fda58ff9700 1 -- 172.21.15.153:0/3404712345 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fda4c010c00 con 0x7fda5c101670 2024-03-11T22:53:40.779 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.137+0000 7fda58ff9700 1 -- 172.21.15.153:0/3404712345 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda4c0215e0 con 0x7fda5c101670 2024-03-11T22:53:40.779 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.137+0000 7fda62428700 1 -- 172.21.15.153:0/3404712345 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda5c113e20 con 0x7fda5c101670 2024-03-11T22:53:40.779 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.138+0000 7fda62428700 1 -- 172.21.15.153:0/3404712345 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda5c114ac0 con 0x7fda5c101670 2024-03-11T22:53:40.779 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.138+0000 7fda58ff9700 1 -- 172.21.15.153:0/3404712345 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7fda4c0223f0 con 0x7fda5c101670 2024-03-11T22:53:40.779 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.139+0000 7fda62428700 1 -- 172.21.15.153:0/3404712345 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda5c05f5c0 con 0x7fda5c101670 2024-03-11T22:53:40.779 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.139+0000 7fda58ff9700 1 --2- 172.21.15.153:0/3404712345 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7fda44036790 0x7fda44038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:40.779 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.139+0000 7fda58ff9700 1 -- 172.21.15.153:0/3404712345 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fda4c014070 con 0x7fda5c101670 2024-03-11T22:53:40.779 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.139+0000 7fda5b7fe700 1 --2- 172.21.15.153:0/3404712345 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7fda44036790 0x7fda44038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:40.779 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.139+0000 7fda5b7fe700 1 --2- 172.21.15.153:0/3404712345 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7fda44036790 0x7fda44038c40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fda50009990 tx=0x7fda50006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:40.779 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.143+0000 7fda58ff9700 1 -- 172.21.15.153:0/3404712345 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7fda4c0226a0 con 0x7fda5c101670 2024-03-11T22:53:40.779 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:39.378+0000 7fda62428700 1 -- 172.21.15.153:0/3404712345 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fda5c04bed0 con 0x7fda5c101670 2024-03-11T22:53:40.779 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:40.229+0000 7fda58ff9700 1 -- 172.21.15.153:0/3404712345 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fda4c021740 con 0x7fda5c101670 2024-03-11T22:53:40.779 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:40.231+0000 7fda58ff9700 1 -- 172.21.15.153:0/3404712345 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fda4c02a9a0 con 0x7fda5c101670 2024-03-11T22:53:40.780 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:40.234+0000 7fda62428700 1 -- 172.21.15.153:0/3404712345 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7fda44036790 msgr2=0x7fda44038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:40.780 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:40.234+0000 7fda62428700 1 --2- 172.21.15.153:0/3404712345 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7fda44036790 0x7fda44038c40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fda50009990 tx=0x7fda50006e40 comp rx=0 tx=0).stop 2024-03-11T22:53:40.780 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:40.234+0000 7fda62428700 1 -- 172.21.15.153:0/3404712345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda5c101670 msgr2=0x7fda5c1136e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:40.780 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:40.234+0000 7fda62428700 1 --2- 172.21.15.153:0/3404712345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda5c101670 0x7fda5c1136e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fda4c006b00 tx=0x7fda4c00b4f0 comp rx=0 tx=0).stop 2024-03-11T22:53:40.780 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:40.234+0000 7fda62428700 1 -- 172.21.15.153:0/3404712345 shutdown_connections 2024-03-11T22:53:40.780 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:40.234+0000 7fda62428700 1 --2- 172.21.15.153:0/3404712345 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7fda44036790 0x7fda44038c40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:40.780 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:40.234+0000 7fda62428700 1 --2- 172.21.15.153:0/3404712345 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda5c101670 0x7fda5c1136e0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:40.780 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:40.234+0000 7fda62428700 1 -- 172.21.15.153:0/3404712345 >> 172.21.15.153:0/3404712345 conn(0x7fda5c0fcdd0 msgr2=0x7fda5c0ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:53:40.780 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:40.235+0000 7fda62428700 1 -- 172.21.15.153:0/3404712345 shutdown_connections 2024-03-11T22:53:40.780 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:40.235+0000 7fda62428700 1 -- 172.21.15.153:0/3404712345 wait complete. 2024-03-11T22:53:41.425 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:41 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/3404712345' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-11T22:53:41.426 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:41 smithi153 ceph-mon[28962]: mgrmap e5: smithi153.nyocep(active, since 12s) 2024-03-11T22:53:42.058 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:53:42.058 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-11T22:53:42.058 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T22:53:42.058 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "active_name": "smithi153.nyocep", 2024-03-11T22:53:42.058 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-11T22:53:42.058 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:53:42.058 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.171+0000 7f7d6ec49700 1 Processor -- start 2024-03-11T22:53:42.059 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.171+0000 7f7d6ec49700 1 -- start start 2024-03-11T22:53:42.059 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.171+0000 7f7d6ec49700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d68102770 0x7f7d68102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:42.059 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.171+0000 7f7d6ec49700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d681030c0 con 0x7f7d68102770 2024-03-11T22:53:42.059 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.171+0000 7f7d6c9e5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d68102770 0x7f7d68102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:42.059 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.171+0000 7f7d6c9e5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d68102770 0x7f7d68102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54924/0 (socket says 172.21.15.153:54924) 2024-03-11T22:53:42.059 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.171+0000 7f7d6c9e5700 1 -- 172.21.15.153:0/3745060032 learned_addr learned my addr 172.21.15.153:0/3745060032 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:53:42.059 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.171+0000 7f7d6c9e5700 1 -- 172.21.15.153:0/3745060032 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d68103200 con 0x7f7d68102770 2024-03-11T22:53:42.059 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.171+0000 7f7d6c9e5700 1 --2- 172.21.15.153:0/3745060032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d68102770 0x7f7d68102b80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c0098e0 tx=0x7f7d5c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c70175114556e688 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:42.059 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.172+0000 7f7d677fe700 1 -- 172.21.15.153:0/3745060032 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d5c00b860 con 0x7f7d68102770 2024-03-11T22:53:42.059 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.172+0000 7f7d677fe700 1 -- 172.21.15.153:0/3745060032 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f7d5c00be50 con 0x7f7d68102770 2024-03-11T22:53:42.059 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.172+0000 7f7d677fe700 1 -- 172.21.15.153:0/3745060032 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d5c010380 con 0x7f7d68102770 2024-03-11T22:53:42.059 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.172+0000 7f7d6ec49700 1 -- 172.21.15.153:0/3745060032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d68102770 msgr2=0x7f7d68102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:42.059 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.172+0000 7f7d6ec49700 1 --2- 172.21.15.153:0/3745060032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d68102770 0x7f7d68102b80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c0098e0 tx=0x7f7d5c009250 comp rx=0 tx=0).stop 2024-03-11T22:53:42.060 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.172+0000 7f7d6ec49700 1 -- 172.21.15.153:0/3745060032 shutdown_connections 2024-03-11T22:53:42.060 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.172+0000 7f7d6ec49700 1 --2- 172.21.15.153:0/3745060032 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d68102770 0x7f7d68102b80 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:42.060 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.172+0000 7f7d6ec49700 1 -- 172.21.15.153:0/3745060032 >> 172.21.15.153:0/3745060032 conn(0x7f7d680fded0 msgr2=0x7f7d68100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:53:42.060 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.173+0000 7f7d6ec49700 1 -- 172.21.15.153:0/3745060032 shutdown_connections 2024-03-11T22:53:42.060 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.173+0000 7f7d6ec49700 1 -- 172.21.15.153:0/3745060032 wait complete. 2024-03-11T22:53:42.060 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.173+0000 7f7d6ec49700 1 Processor -- start 2024-03-11T22:53:42.060 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.173+0000 7f7d6ec49700 1 -- start start 2024-03-11T22:53:42.060 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.173+0000 7f7d6ec49700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d68102770 0x7f7d68111500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:42.060 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.173+0000 7f7d6ec49700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d68111a40 con 0x7f7d68102770 2024-03-11T22:53:42.060 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.173+0000 7f7d6c9e5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d68102770 0x7f7d68111500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:42.060 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.173+0000 7f7d6c9e5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d68102770 0x7f7d68111500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54932/0 (socket says 172.21.15.153:54932) 2024-03-11T22:53:42.060 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.173+0000 7f7d6c9e5700 1 -- 172.21.15.153:0/2306953359 learned_addr learned my addr 172.21.15.153:0/2306953359 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:53:42.060 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.174+0000 7f7d6c9e5700 1 -- 172.21.15.153:0/2306953359 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d5c0094f0 con 0x7f7d68102770 2024-03-11T22:53:42.060 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.174+0000 7f7d6c9e5700 1 --2- 172.21.15.153:0/2306953359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d68102770 0x7f7d68111500 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c00b440 tx=0x7f7d5c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:42.060 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.174+0000 7f7d657fa700 1 -- 172.21.15.153:0/2306953359 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d5c0107c0 con 0x7f7d68102770 2024-03-11T22:53:42.061 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.174+0000 7f7d657fa700 1 -- 172.21.15.153:0/2306953359 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f7d5c010db0 con 0x7f7d68102770 2024-03-11T22:53:42.061 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.174+0000 7f7d6ec49700 1 -- 172.21.15.153:0/2306953359 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d68111c40 con 0x7f7d68102770 2024-03-11T22:53:42.061 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.174+0000 7f7d657fa700 1 -- 172.21.15.153:0/2306953359 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d5c01ada0 con 0x7f7d68102770 2024-03-11T22:53:42.061 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.174+0000 7f7d6ec49700 1 -- 172.21.15.153:0/2306953359 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d681128e0 con 0x7f7d68102770 2024-03-11T22:53:42.061 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.175+0000 7f7d657fa700 1 -- 172.21.15.153:0/2306953359 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f7d5c024480 con 0x7f7d68102770 2024-03-11T22:53:42.061 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.175+0000 7f7d6ec49700 1 -- 172.21.15.153:0/2306953359 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d6805f5c0 con 0x7f7d68102770 2024-03-11T22:53:42.061 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.175+0000 7f7d657fa700 1 --2- 172.21.15.153:0/2306953359 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7f7d58036790 0x7f7d58038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:42.061 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.175+0000 7f7d657fa700 1 -- 172.21.15.153:0/2306953359 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f7d5c04ace0 con 0x7f7d68102770 2024-03-11T22:53:42.061 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.175+0000 7f7d67fff700 1 -- 172.21.15.153:0/2306953359 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7f7d58036790 msgr2=0x7f7d58038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/2578439373 2024-03-11T22:53:42.061 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.175+0000 7f7d67fff700 1 --2- 172.21.15.153:0/2306953359 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7f7d58036790 0x7f7d58038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T22:53:42.061 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.180+0000 7f7d657fa700 1 -- 172.21.15.153:0/2306953359 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f7d5c018350 con 0x7f7d68102770 2024-03-11T22:53:42.061 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.376+0000 7f7d67fff700 1 -- 172.21.15.153:0/2306953359 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7f7d58036790 msgr2=0x7f7d58038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/2578439373 2024-03-11T22:53:42.061 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.376+0000 7f7d67fff700 1 --2- 172.21.15.153:0/2306953359 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7f7d58036790 0x7f7d58038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-11T22:53:42.061 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.420+0000 7f7d6ec49700 1 -- 172.21.15.153:0/2306953359 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f7d68112c20 con 0x7f7d68102770 2024-03-11T22:53:42.062 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.421+0000 7f7d657fa700 1 -- 172.21.15.153:0/2306953359 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f7d5c02cb80 con 0x7f7d68102770 2024-03-11T22:53:42.062 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.424+0000 7f7d6ec49700 1 -- 172.21.15.153:0/2306953359 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7f7d58036790 msgr2=0x7f7d58038c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T22:53:42.062 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.424+0000 7f7d6ec49700 1 --2- 172.21.15.153:0/2306953359 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7f7d58036790 0x7f7d58038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:42.062 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.424+0000 7f7d6ec49700 1 -- 172.21.15.153:0/2306953359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d68102770 msgr2=0x7f7d68111500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:42.062 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.424+0000 7f7d6ec49700 1 --2- 172.21.15.153:0/2306953359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d68102770 0x7f7d68111500 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c00b440 tx=0x7f7d5c009250 comp rx=0 tx=0).stop 2024-03-11T22:53:42.062 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.424+0000 7f7d6ec49700 1 -- 172.21.15.153:0/2306953359 shutdown_connections 2024-03-11T22:53:42.062 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.424+0000 7f7d6ec49700 1 --2- 172.21.15.153:0/2306953359 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7f7d58036790 0x7f7d58038c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:42.062 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.424+0000 7f7d6ec49700 1 --2- 172.21.15.153:0/2306953359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7d68102770 0x7f7d68111500 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:42.062 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.424+0000 7f7d6ec49700 1 -- 172.21.15.153:0/2306953359 >> 172.21.15.153:0/2306953359 conn(0x7f7d680fded0 msgr2=0x7f7d680fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:53:42.062 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.424+0000 7f7d6ec49700 1 -- 172.21.15.153:0/2306953359 shutdown_connections 2024-03-11T22:53:42.062 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:41.424+0000 7f7d6ec49700 1 -- 172.21.15.153:0/2306953359 wait complete. 2024-03-11T22:53:42.062 INFO:teuthology.orchestra.run.smithi153.stdout:Waiting for the mgr to restart... 2024-03-11T22:53:42.062 INFO:teuthology.orchestra.run.smithi153.stdout:Waiting for mgr epoch 5... 2024-03-11T22:53:42.413 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:42 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/2306953359' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-11T22:53:47.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:46 smithi153 ceph-mon[28962]: Active manager daemon smithi153.nyocep restarted 2024-03-11T22:53:47.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:46 smithi153 ceph-mon[28962]: Activating manager daemon smithi153.nyocep 2024-03-11T22:53:47.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:46 smithi153 ceph-mon[28962]: osdmap e2: 0 total, 0 up, 0 in 2024-03-11T22:53:47.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:46 smithi153 ceph-mon[28962]: mgrmap e6: smithi153.nyocep(active, starting, since 0.00329082s) 2024-03-11T22:53:47.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:46 smithi153 ceph-mon[28962]: from='mgr.14118 172.21.15.153:0/870689843' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-03-11T22:53:47.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:46 smithi153 ceph-mon[28962]: from='mgr.14118 172.21.15.153:0/870689843' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mgr metadata", "who": "smithi153.nyocep", "id": "smithi153.nyocep"}]: dispatch 2024-03-11T22:53:47.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:46 smithi153 ceph-mon[28962]: from='mgr.14118 172.21.15.153:0/870689843' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:53:47.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:46 smithi153 ceph-mon[28962]: from='mgr.14118 172.21.15.153:0/870689843' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:53:47.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:46 smithi153 ceph-mon[28962]: from='mgr.14118 172.21.15.153:0/870689843' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:53:47.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:46 smithi153 ceph-mon[28962]: Manager daemon smithi153.nyocep is now available 2024-03-11T22:53:47.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:46 smithi153 ceph-mon[28962]: from='mgr.14118 172.21.15.153:0/870689843' entity='mgr.smithi153.nyocep' 2024-03-11T22:53:47.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:46 smithi153 ceph-mon[28962]: from='mgr.14118 172.21.15.153:0/870689843' entity='mgr.smithi153.nyocep' 2024-03-11T22:53:47.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:46 smithi153 ceph-mon[28962]: from='mgr.14118 172.21.15.153:0/870689843' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:53:47.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:46 smithi153 ceph-mon[28962]: from='mgr.14118 172.21.15.153:0/870689843' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:53:47.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:46 smithi153 ceph-mon[28962]: from='mgr.14118 172.21.15.153:0/870689843' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:53:48.143 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:47 smithi153 ceph-mon[28962]: Found migration_current of "None". Setting to last migration. 2024-03-11T22:53:48.143 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:47 smithi153 ceph-mon[28962]: from='mgr.14118 172.21.15.153:0/870689843' entity='mgr.smithi153.nyocep' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.nyocep/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:53:48.143 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:47 smithi153 ceph-mon[28962]: from='mgr.14118 172.21.15.153:0/870689843' entity='mgr.smithi153.nyocep' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.nyocep/trash_purge_schedule"}]: dispatch 2024-03-11T22:53:48.144 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:47 smithi153 ceph-mon[28962]: mgrmap e7: smithi153.nyocep(active, since 1.00516s) 2024-03-11T22:53:48.596 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:53:48.596 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-11T22:53:48.596 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-11T22:53:48.596 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:53:48.596 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.359+0000 7f670c673700 1 Processor -- start 2024-03-11T22:53:48.596 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.359+0000 7f670c673700 1 -- start start 2024-03-11T22:53:48.596 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.360+0000 7f670c673700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6704102770 0x7f6704102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:48.596 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.360+0000 7f670c673700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67041030c0 con 0x7f6704102770 2024-03-11T22:53:48.596 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.360+0000 7f670a40f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6704102770 0x7f6704102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:48.596 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.360+0000 7f670a40f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6704102770 0x7f6704102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:51478/0 (socket says 172.21.15.153:51478) 2024-03-11T22:53:48.596 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.360+0000 7f670a40f700 1 -- 172.21.15.153:0/3184330555 learned_addr learned my addr 172.21.15.153:0/3184330555 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:53:48.597 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.360+0000 7f670a40f700 1 -- 172.21.15.153:0/3184330555 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6704103200 con 0x7f6704102770 2024-03-11T22:53:48.597 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.360+0000 7f670a40f700 1 --2- 172.21.15.153:0/3184330555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6704102770 0x7f6704102b80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f67000098e0 tx=0x7f6700009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a61f5a341050f12 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:48.597 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.361+0000 7f670940d700 1 -- 172.21.15.153:0/3184330555 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f670000b860 con 0x7f6704102770 2024-03-11T22:53:48.597 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.361+0000 7f670940d700 1 -- 172.21.15.153:0/3184330555 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f670000be50 con 0x7f6704102770 2024-03-11T22:53:48.597 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.361+0000 7f670940d700 1 -- 172.21.15.153:0/3184330555 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6700010380 con 0x7f6704102770 2024-03-11T22:53:48.597 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.361+0000 7f670c673700 1 -- 172.21.15.153:0/3184330555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6704102770 msgr2=0x7f6704102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:48.597 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.361+0000 7f670c673700 1 --2- 172.21.15.153:0/3184330555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6704102770 0x7f6704102b80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f67000098e0 tx=0x7f6700009250 comp rx=0 tx=0).stop 2024-03-11T22:53:48.597 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.362+0000 7f670c673700 1 -- 172.21.15.153:0/3184330555 shutdown_connections 2024-03-11T22:53:48.597 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.362+0000 7f670c673700 1 --2- 172.21.15.153:0/3184330555 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6704102770 0x7f6704102b80 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:48.597 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.362+0000 7f670c673700 1 -- 172.21.15.153:0/3184330555 >> 172.21.15.153:0/3184330555 conn(0x7f67040fded0 msgr2=0x7f6704100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:53:48.597 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.362+0000 7f670c673700 1 -- 172.21.15.153:0/3184330555 shutdown_connections 2024-03-11T22:53:48.597 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.362+0000 7f670c673700 1 -- 172.21.15.153:0/3184330555 wait complete. 2024-03-11T22:53:48.597 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.362+0000 7f670c673700 1 Processor -- start 2024-03-11T22:53:48.597 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.362+0000 7f670c673700 1 -- start start 2024-03-11T22:53:48.597 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.362+0000 7f670c673700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6704102770 0x7f6704111500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:48.598 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.362+0000 7f670c673700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6704111a40 con 0x7f6704102770 2024-03-11T22:53:48.598 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.362+0000 7f670a40f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6704102770 0x7f6704111500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:48.598 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.362+0000 7f670a40f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6704102770 0x7f6704111500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:51492/0 (socket says 172.21.15.153:51492) 2024-03-11T22:53:48.598 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.362+0000 7f670a40f700 1 -- 172.21.15.153:0/1448704145 learned_addr learned my addr 172.21.15.153:0/1448704145 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:53:48.598 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.363+0000 7f670a40f700 1 -- 172.21.15.153:0/1448704145 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67000094f0 con 0x7f6704102770 2024-03-11T22:53:48.598 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.363+0000 7f670a40f700 1 --2- 172.21.15.153:0/1448704145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6704102770 0x7f6704111500 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f670000b440 tx=0x7f6700009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:48.598 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.363+0000 7f66faffd700 1 -- 172.21.15.153:0/1448704145 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67000107c0 con 0x7f6704102770 2024-03-11T22:53:48.598 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.363+0000 7f66faffd700 1 -- 172.21.15.153:0/1448704145 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f6700010db0 con 0x7f6704102770 2024-03-11T22:53:48.598 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.363+0000 7f66faffd700 1 -- 172.21.15.153:0/1448704145 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f670001ada0 con 0x7f6704102770 2024-03-11T22:53:48.598 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.363+0000 7f670c673700 1 -- 172.21.15.153:0/1448704145 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6704111c40 con 0x7f6704102770 2024-03-11T22:53:48.598 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.363+0000 7f670c673700 1 -- 172.21.15.153:0/1448704145 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67041128e0 con 0x7f6704102770 2024-03-11T22:53:48.598 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.364+0000 7f66faffd700 1 -- 172.21.15.153:0/1448704145 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f670002e440 con 0x7f6704102770 2024-03-11T22:53:48.598 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.364+0000 7f66faffd700 1 --2- 172.21.15.153:0/1448704145 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7f66f00367e0 0x7f66f0038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:48.598 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.364+0000 7f66faffd700 1 -- 172.21.15.153:0/1448704145 --> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66f00393a0 con 0x7f66f00367e0 2024-03-11T22:53:48.599 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.364+0000 7f66faffd700 1 -- 172.21.15.153:0/1448704145 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f670004ae60 con 0x7f6704102770 2024-03-11T22:53:48.599 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.364+0000 7f6709c0e700 1 -- 172.21.15.153:0/1448704145 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7f66f00367e0 msgr2=0x7f66f0038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/2578439373 2024-03-11T22:53:48.599 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.364+0000 7f6709c0e700 1 --2- 172.21.15.153:0/1448704145 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7f66f00367e0 0x7f66f0038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T22:53:48.599 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.565+0000 7f6709c0e700 1 -- 172.21.15.153:0/1448704145 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7f66f00367e0 msgr2=0x7f66f0038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/2578439373 2024-03-11T22:53:48.599 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.565+0000 7f6709c0e700 1 --2- 172.21.15.153:0/1448704145 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7f66f00367e0 0x7f66f0038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-11T22:53:48.599 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.966+0000 7f6709c0e700 1 -- 172.21.15.153:0/1448704145 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7f66f00367e0 msgr2=0x7f66f0038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/2578439373 2024-03-11T22:53:48.599 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:43.966+0000 7f6709c0e700 1 --2- 172.21.15.153:0/1448704145 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7f66f00367e0 0x7f66f0038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-11T22:53:48.599 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:44.767+0000 7f6709c0e700 1 -- 172.21.15.153:0/1448704145 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7f66f00367e0 msgr2=0x7f66f0038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/2578439373 2024-03-11T22:53:48.599 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:44.767+0000 7f6709c0e700 1 --2- 172.21.15.153:0/1448704145 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7f66f00367e0 0x7f66f0038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-11T22:53:48.599 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:46.368+0000 7f6709c0e700 1 -- 172.21.15.153:0/1448704145 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7f66f00367e0 msgr2=0x7f66f0038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/2578439373 2024-03-11T22:53:48.599 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:46.368+0000 7f6709c0e700 1 --2- 172.21.15.153:0/1448704145 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7f66f00367e0 0x7f66f0038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-11T22:53:48.599 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:46.837+0000 7f66faffd700 1 -- 172.21.15.153:0/1448704145 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mgrmap(e 6) v1 ==== 43135+0+0 (secure 0 0 0) 0x7f670002ecf0 con 0x7f6704102770 2024-03-11T22:53:48.599 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:46.837+0000 7f66faffd700 1 -- 172.21.15.153:0/1448704145 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7f66f00367e0 msgr2=0x7f66f0038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T22:53:48.599 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:46.837+0000 7f66faffd700 1 --2- 172.21.15.153:0/1448704145 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7f66f00367e0 0x7f66f0038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:48.600 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:47.837+0000 7f66faffd700 1 -- 172.21.15.153:0/1448704145 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mgrmap(e 7) v1 ==== 43262+0+0 (secure 0 0 0) 0x7f670004c880 con 0x7f6704102770 2024-03-11T22:53:48.600 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:47.838+0000 7f66faffd700 1 --2- 172.21.15.153:0/1448704145 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f66f0039fc0 0x7f66f003c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:48.600 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:47.838+0000 7f66faffd700 1 -- 172.21.15.153:0/1448704145 --> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66f00393a0 con 0x7f66f0039fc0 2024-03-11T22:53:48.600 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:47.838+0000 7f6709c0e700 1 --2- 172.21.15.153:0/1448704145 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f66f0039fc0 0x7f66f003c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:48.600 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:47.839+0000 7f6709c0e700 1 --2- 172.21.15.153:0/1448704145 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f66f0039fc0 0x7f66f003c3a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f66f40039d0 tx=0x7f66f4009350 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:48.600 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:47.839+0000 7f66faffd700 1 -- 172.21.15.153:0/1448704145 <== mgr.14118 v2:172.21.15.153:6800/1518383615 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f66f00393a0 con 0x7f66f0039fc0 2024-03-11T22:53:48.600 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:47.845+0000 7f670c673700 1 -- 172.21.15.153:0/1448704145 --> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f67041120d0 con 0x7f66f0039fc0 2024-03-11T22:53:48.600 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:47.845+0000 7f66faffd700 1 -- 172.21.15.153:0/1448704145 <== mgr.14118 v2:172.21.15.153:6800/1518383615 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f67041120d0 con 0x7f66f0039fc0 2024-03-11T22:53:48.600 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:47.846+0000 7f670c673700 1 -- 172.21.15.153:0/1448704145 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f66f0039fc0 msgr2=0x7f66f003c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:48.600 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:47.846+0000 7f670c673700 1 --2- 172.21.15.153:0/1448704145 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f66f0039fc0 0x7f66f003c3a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f66f40039d0 tx=0x7f66f4009350 comp rx=0 tx=0).stop 2024-03-11T22:53:48.600 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:47.846+0000 7f670c673700 1 -- 172.21.15.153:0/1448704145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6704102770 msgr2=0x7f6704111500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:48.600 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:47.846+0000 7f670c673700 1 --2- 172.21.15.153:0/1448704145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6704102770 0x7f6704111500 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f670000b440 tx=0x7f6700009250 comp rx=0 tx=0).stop 2024-03-11T22:53:48.600 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:47.846+0000 7f670c673700 1 -- 172.21.15.153:0/1448704145 shutdown_connections 2024-03-11T22:53:48.600 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:47.846+0000 7f670c673700 1 --2- 172.21.15.153:0/1448704145 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f66f0039fc0 0x7f66f003c3a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:48.600 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:47.846+0000 7f670c673700 1 --2- 172.21.15.153:0/1448704145 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6704102770 0x7f6704111500 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:48.601 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:47.846+0000 7f670c673700 1 --2- 172.21.15.153:0/1448704145 >> [v2:172.21.15.153:6800/2578439373,v1:172.21.15.153:6801/2578439373] conn(0x7f66f00367e0 0x7f66f0038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:48.601 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:47.846+0000 7f670c673700 1 -- 172.21.15.153:0/1448704145 >> 172.21.15.153:0/1448704145 conn(0x7f67040fded0 msgr2=0x7f67040fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:53:48.601 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:47.846+0000 7f670c673700 1 -- 172.21.15.153:0/1448704145 shutdown_connections 2024-03-11T22:53:48.601 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:47.846+0000 7f670c673700 1 -- 172.21.15.153:0/1448704145 wait complete. 2024-03-11T22:53:48.601 INFO:teuthology.orchestra.run.smithi153.stdout:mgr epoch 5 is available 2024-03-11T22:53:48.601 INFO:teuthology.orchestra.run.smithi153.stdout:Setting orchestrator backend to cephadm... 2024-03-11T22:53:49.183 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:48 smithi153 ceph-mon[28962]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-11T22:53:49.184 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:48 smithi153 ceph-mon[28962]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-11T22:53:49.184 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:48 smithi153 ceph-mon[28962]: from='mgr.14118 172.21.15.153:0/870689843' entity='mgr.smithi153.nyocep' 2024-03-11T22:53:49.184 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:48 smithi153 ceph-mon[28962]: from='mgr.14118 172.21.15.153:0/870689843' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:53:49.734 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.972+0000 7f18c048e700 1 Processor -- start 2024-03-11T22:53:49.734 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.973+0000 7f18c048e700 1 -- start start 2024-03-11T22:53:49.735 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.973+0000 7f18c048e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18b8102770 0x7f18b8102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:49.735 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.973+0000 7f18c048e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b81030c0 con 0x7f18b8102770 2024-03-11T22:53:49.735 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.973+0000 7f18be22a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18b8102770 0x7f18b8102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:49.735 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.973+0000 7f18be22a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18b8102770 0x7f18b8102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:51564/0 (socket says 172.21.15.153:51564) 2024-03-11T22:53:49.735 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.973+0000 7f18be22a700 1 -- 172.21.15.153:0/160617965 learned_addr learned my addr 172.21.15.153:0/160617965 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:53:49.735 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.973+0000 7f18be22a700 1 -- 172.21.15.153:0/160617965 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b8103200 con 0x7f18b8102770 2024-03-11T22:53:49.735 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.973+0000 7f18be22a700 1 --2- 172.21.15.153:0/160617965 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18b8102770 0x7f18b8102b80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f18b40098e0 tx=0x7f18b4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63ff2d51fbf5be85 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:49.735 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.974+0000 7f18bd228700 1 -- 172.21.15.153:0/160617965 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18b400b860 con 0x7f18b8102770 2024-03-11T22:53:49.735 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.974+0000 7f18bd228700 1 -- 172.21.15.153:0/160617965 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f18b400be50 con 0x7f18b8102770 2024-03-11T22:53:49.735 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.974+0000 7f18bd228700 1 -- 172.21.15.153:0/160617965 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18b4010380 con 0x7f18b8102770 2024-03-11T22:53:49.735 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.974+0000 7f18c048e700 1 -- 172.21.15.153:0/160617965 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18b8102770 msgr2=0x7f18b8102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:49.735 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.974+0000 7f18c048e700 1 --2- 172.21.15.153:0/160617965 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18b8102770 0x7f18b8102b80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f18b40098e0 tx=0x7f18b4009250 comp rx=0 tx=0).stop 2024-03-11T22:53:49.735 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.974+0000 7f18c048e700 1 -- 172.21.15.153:0/160617965 shutdown_connections 2024-03-11T22:53:49.736 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.974+0000 7f18c048e700 1 --2- 172.21.15.153:0/160617965 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18b8102770 0x7f18b8102b80 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:49.736 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.974+0000 7f18c048e700 1 -- 172.21.15.153:0/160617965 >> 172.21.15.153:0/160617965 conn(0x7f18b80fded0 msgr2=0x7f18b8100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:53:49.736 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.974+0000 7f18c048e700 1 -- 172.21.15.153:0/160617965 shutdown_connections 2024-03-11T22:53:49.736 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.974+0000 7f18c048e700 1 -- 172.21.15.153:0/160617965 wait complete. 2024-03-11T22:53:49.736 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.975+0000 7f18c048e700 1 Processor -- start 2024-03-11T22:53:49.736 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.975+0000 7f18c048e700 1 -- start start 2024-03-11T22:53:49.736 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.975+0000 7f18c048e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18b8102770 0x7f18b8111590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:49.736 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.975+0000 7f18c048e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b8111ad0 con 0x7f18b8102770 2024-03-11T22:53:49.736 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.975+0000 7f18be22a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18b8102770 0x7f18b8111590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:49.736 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.975+0000 7f18be22a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18b8102770 0x7f18b8111590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:51566/0 (socket says 172.21.15.153:51566) 2024-03-11T22:53:49.736 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.975+0000 7f18be22a700 1 -- 172.21.15.153:0/3614053768 learned_addr learned my addr 172.21.15.153:0/3614053768 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:53:49.736 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.975+0000 7f18be22a700 1 -- 172.21.15.153:0/3614053768 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b40094f0 con 0x7f18b8102770 2024-03-11T22:53:49.736 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.975+0000 7f18be22a700 1 --2- 172.21.15.153:0/3614053768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18b8102770 0x7f18b8111590 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f18b400b440 tx=0x7f18b4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:49.736 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.976+0000 7f18aeffd700 1 -- 172.21.15.153:0/3614053768 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18b4010760 con 0x7f18b8102770 2024-03-11T22:53:49.736 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.976+0000 7f18c048e700 1 -- 172.21.15.153:0/3614053768 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18b8111cd0 con 0x7f18b8102770 2024-03-11T22:53:49.737 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.976+0000 7f18aeffd700 1 -- 172.21.15.153:0/3614053768 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f18b4010d50 con 0x7f18b8102770 2024-03-11T22:53:49.737 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.976+0000 7f18aeffd700 1 -- 172.21.15.153:0/3614053768 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18b401ae20 con 0x7f18b8102770 2024-03-11T22:53:49.737 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.976+0000 7f18c048e700 1 -- 172.21.15.153:0/3614053768 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18b8112970 con 0x7f18b8102770 2024-03-11T22:53:49.737 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.976+0000 7f18aeffd700 1 -- 172.21.15.153:0/3614053768 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 7) v1 ==== 43262+0+0 (secure 0 0 0) 0x7f18b402e440 con 0x7f18b8102770 2024-03-11T22:53:49.737 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.977+0000 7f18aeffd700 1 --2- 172.21.15.153:0/3614053768 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f18a4036660 0x7f18a4038b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:49.737 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.977+0000 7f18aeffd700 1 -- 172.21.15.153:0/3614053768 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f18b404a430 con 0x7f18b8102770 2024-03-11T22:53:49.737 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.977+0000 7f18c048e700 1 -- 172.21.15.153:0/3614053768 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18b805f5c0 con 0x7f18b8102770 2024-03-11T22:53:49.737 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.977+0000 7f18bda29700 1 --2- 172.21.15.153:0/3614053768 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f18a4036660 0x7f18a4038b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:49.737 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.977+0000 7f18bda29700 1 --2- 172.21.15.153:0/3614053768 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f18a4036660 0x7f18a4038b10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f18a8009990 tx=0x7f18a8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:49.737 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:48.982+0000 7f18aeffd700 1 -- 172.21.15.153:0/3614053768 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f18b4018940 con 0x7f18b8102770 2024-03-11T22:53:49.737 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:49.178+0000 7f18c048e700 1 -- 172.21.15.153:0/3614053768 --> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f18b81001b0 con 0x7f18a4036660 2024-03-11T22:53:49.737 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:49.184+0000 7f18aeffd700 1 -- 172.21.15.153:0/3614053768 <== mgr.14118 v2:172.21.15.153:6800/1518383615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f18b81001b0 con 0x7f18a4036660 2024-03-11T22:53:49.737 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:49.187+0000 7f18c048e700 1 -- 172.21.15.153:0/3614053768 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f18a4036660 msgr2=0x7f18a4038b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:49.737 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:49.187+0000 7f18c048e700 1 --2- 172.21.15.153:0/3614053768 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f18a4036660 0x7f18a4038b10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f18a8009990 tx=0x7f18a8006e40 comp rx=0 tx=0).stop 2024-03-11T22:53:49.737 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:49.187+0000 7f18c048e700 1 -- 172.21.15.153:0/3614053768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18b8102770 msgr2=0x7f18b8111590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:49.738 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:49.187+0000 7f18c048e700 1 --2- 172.21.15.153:0/3614053768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18b8102770 0x7f18b8111590 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f18b400b440 tx=0x7f18b4009250 comp rx=0 tx=0).stop 2024-03-11T22:53:49.738 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:49.187+0000 7f18c048e700 1 -- 172.21.15.153:0/3614053768 shutdown_connections 2024-03-11T22:53:49.738 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:49.187+0000 7f18c048e700 1 --2- 172.21.15.153:0/3614053768 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f18a4036660 0x7f18a4038b10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:49.738 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:49.187+0000 7f18c048e700 1 --2- 172.21.15.153:0/3614053768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18b8102770 0x7f18b8111590 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:49.738 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:49.187+0000 7f18c048e700 1 -- 172.21.15.153:0/3614053768 >> 172.21.15.153:0/3614053768 conn(0x7f18b80fded0 msgr2=0x7f18b80ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:53:49.738 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:49.187+0000 7f18c048e700 1 -- 172.21.15.153:0/3614053768 shutdown_connections 2024-03-11T22:53:49.738 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:49.187+0000 7f18c048e700 1 -- 172.21.15.153:0/3614053768 wait complete. 2024-03-11T22:53:50.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:50 smithi153 ceph-mon[28962]: [11/Mar/2024:22:53:48] ENGINE Bus STARTING 2024-03-11T22:53:50.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:50 smithi153 ceph-mon[28962]: [11/Mar/2024:22:53:48] ENGINE Serving on https://172.21.15.153:7150 2024-03-11T22:53:50.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:50 smithi153 ceph-mon[28962]: [11/Mar/2024:22:53:48] ENGINE Bus STARTED 2024-03-11T22:53:50.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:50 smithi153 ceph-mon[28962]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:53:50.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:50 smithi153 ceph-mon[28962]: from='mgr.14118 172.21.15.153:0/870689843' entity='mgr.smithi153.nyocep' 2024-03-11T22:53:50.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:50 smithi153 ceph-mon[28962]: from='mgr.14118 172.21.15.153:0/870689843' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:53:50.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:50 smithi153 ceph-mon[28962]: mgrmap e8: smithi153.nyocep(active, since 2s) 2024-03-11T22:53:50.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-11T22:53:50.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.140+0000 7ff9deb26700 1 Processor -- start 2024-03-11T22:53:50.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.140+0000 7ff9deb26700 1 -- start start 2024-03-11T22:53:50.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.140+0000 7ff9deb26700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9d8101670 0x7ff9d8101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:50.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.140+0000 7ff9deb26700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d8101fc0 con 0x7ff9d8101670 2024-03-11T22:53:50.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.140+0000 7ff9dc8c2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9d8101670 0x7ff9d8101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:50.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.140+0000 7ff9dc8c2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9d8101670 0x7ff9d8101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:51576/0 (socket says 172.21.15.153:51576) 2024-03-11T22:53:50.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.140+0000 7ff9dc8c2700 1 -- 172.21.15.153:0/1022235839 learned_addr learned my addr 172.21.15.153:0/1022235839 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:53:50.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.141+0000 7ff9dc8c2700 1 -- 172.21.15.153:0/1022235839 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d8102100 con 0x7ff9d8101670 2024-03-11T22:53:50.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.141+0000 7ff9dc8c2700 1 --2- 172.21.15.153:0/1022235839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9d8101670 0x7ff9d8101a80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc0098e0 tx=0x7ff9cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42cb34e24ace3a70 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:50.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.141+0000 7ff9d77fe700 1 -- 172.21.15.153:0/1022235839 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9cc00b860 con 0x7ff9d8101670 2024-03-11T22:53:50.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.141+0000 7ff9d77fe700 1 -- 172.21.15.153:0/1022235839 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff9cc00be50 con 0x7ff9d8101670 2024-03-11T22:53:50.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.141+0000 7ff9d77fe700 1 -- 172.21.15.153:0/1022235839 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9cc010380 con 0x7ff9d8101670 2024-03-11T22:53:50.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.141+0000 7ff9deb26700 1 -- 172.21.15.153:0/1022235839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9d8101670 msgr2=0x7ff9d8101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:50.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.141+0000 7ff9deb26700 1 --2- 172.21.15.153:0/1022235839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9d8101670 0x7ff9d8101a80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc0098e0 tx=0x7ff9cc009250 comp rx=0 tx=0).stop 2024-03-11T22:53:50.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.141+0000 7ff9deb26700 1 -- 172.21.15.153:0/1022235839 shutdown_connections 2024-03-11T22:53:50.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.141+0000 7ff9deb26700 1 --2- 172.21.15.153:0/1022235839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9d8101670 0x7ff9d8101a80 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:50.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.141+0000 7ff9deb26700 1 -- 172.21.15.153:0/1022235839 >> 172.21.15.153:0/1022235839 conn(0x7ff9d80fcdd0 msgr2=0x7ff9d80ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:53:50.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.142+0000 7ff9deb26700 1 -- 172.21.15.153:0/1022235839 shutdown_connections 2024-03-11T22:53:50.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.142+0000 7ff9deb26700 1 -- 172.21.15.153:0/1022235839 wait complete. 2024-03-11T22:53:50.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.142+0000 7ff9deb26700 1 Processor -- start 2024-03-11T22:53:50.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.142+0000 7ff9deb26700 1 -- start start 2024-03-11T22:53:50.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.142+0000 7ff9deb26700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9d8101670 0x7ff9d8113a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:50.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.142+0000 7ff9deb26700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d8113f50 con 0x7ff9d8101670 2024-03-11T22:53:50.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.142+0000 7ff9dc8c2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9d8101670 0x7ff9d8113a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:50.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.142+0000 7ff9dc8c2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9d8101670 0x7ff9d8113a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:51592/0 (socket says 172.21.15.153:51592) 2024-03-11T22:53:50.909 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.142+0000 7ff9dc8c2700 1 -- 172.21.15.153:0/4194622227 learned_addr learned my addr 172.21.15.153:0/4194622227 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:53:50.909 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.143+0000 7ff9dc8c2700 1 -- 172.21.15.153:0/4194622227 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9cc0094f0 con 0x7ff9d8101670 2024-03-11T22:53:50.909 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.143+0000 7ff9dc8c2700 1 --2- 172.21.15.153:0/4194622227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9d8101670 0x7ff9d8113a10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc00b440 tx=0x7ff9cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:50.909 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.143+0000 7ff9d57fa700 1 -- 172.21.15.153:0/4194622227 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9cc010760 con 0x7ff9d8101670 2024-03-11T22:53:50.909 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.143+0000 7ff9d57fa700 1 -- 172.21.15.153:0/4194622227 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff9cc010d50 con 0x7ff9d8101670 2024-03-11T22:53:50.909 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.143+0000 7ff9deb26700 1 -- 172.21.15.153:0/4194622227 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9d8114150 con 0x7ff9d8101670 2024-03-11T22:53:50.909 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.143+0000 7ff9d57fa700 1 -- 172.21.15.153:0/4194622227 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9cc024d80 con 0x7ff9d8101670 2024-03-11T22:53:50.909 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.143+0000 7ff9deb26700 1 -- 172.21.15.153:0/4194622227 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9d8114df0 con 0x7ff9d8101670 2024-03-11T22:53:50.909 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.144+0000 7ff9d57fa700 1 -- 172.21.15.153:0/4194622227 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7ff9cc01a430 con 0x7ff9d8101670 2024-03-11T22:53:50.909 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.144+0000 7ff9d57fa700 1 --2- 172.21.15.153:0/4194622227 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7ff9c80367d0 0x7ff9c8038c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:50.909 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.144+0000 7ff9deb26700 1 -- 172.21.15.153:0/4194622227 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9d805f5c0 con 0x7ff9d8101670 2024-03-11T22:53:50.909 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.144+0000 7ff9d57fa700 1 -- 172.21.15.153:0/4194622227 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7ff9cc04a670 con 0x7ff9d8101670 2024-03-11T22:53:50.909 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.144+0000 7ff9d7fff700 1 --2- 172.21.15.153:0/4194622227 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7ff9c80367d0 0x7ff9c8038c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:50.910 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.145+0000 7ff9d7fff700 1 --2- 172.21.15.153:0/4194622227 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7ff9c80367d0 0x7ff9c8038c80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff9c4009990 tx=0x7ff9c4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:50.910 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.149+0000 7ff9d57fa700 1 -- 172.21.15.153:0/4194622227 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7ff9cc01a6e0 con 0x7ff9d8101670 2024-03-11T22:53:50.910 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.335+0000 7ff9deb26700 1 -- 172.21.15.153:0/4194622227 --> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7ff9d81035f0 con 0x7ff9c80367d0 2024-03-11T22:53:50.910 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.336+0000 7ff9d57fa700 1 -- 172.21.15.153:0/4194622227 <== mgr.14118 v2:172.21.15.153:6800/1518383615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7ff9d81035f0 con 0x7ff9c80367d0 2024-03-11T22:53:50.910 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.338+0000 7ff9deb26700 1 -- 172.21.15.153:0/4194622227 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7ff9c80367d0 msgr2=0x7ff9c8038c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:50.910 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.338+0000 7ff9deb26700 1 --2- 172.21.15.153:0/4194622227 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7ff9c80367d0 0x7ff9c8038c80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff9c4009990 tx=0x7ff9c4006e40 comp rx=0 tx=0).stop 2024-03-11T22:53:50.910 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.338+0000 7ff9deb26700 1 -- 172.21.15.153:0/4194622227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9d8101670 msgr2=0x7ff9d8113a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:50.910 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.338+0000 7ff9deb26700 1 --2- 172.21.15.153:0/4194622227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9d8101670 0x7ff9d8113a10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc00b440 tx=0x7ff9cc009250 comp rx=0 tx=0).stop 2024-03-11T22:53:50.910 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.338+0000 7ff9deb26700 1 -- 172.21.15.153:0/4194622227 shutdown_connections 2024-03-11T22:53:50.910 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.338+0000 7ff9deb26700 1 --2- 172.21.15.153:0/4194622227 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7ff9c80367d0 0x7ff9c8038c80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:50.910 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.338+0000 7ff9deb26700 1 --2- 172.21.15.153:0/4194622227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9d8101670 0x7ff9d8113a10 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:50.910 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.338+0000 7ff9deb26700 1 -- 172.21.15.153:0/4194622227 >> 172.21.15.153:0/4194622227 conn(0x7ff9d80fcdd0 msgr2=0x7ff9d81063e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:53:50.910 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.338+0000 7ff9deb26700 1 -- 172.21.15.153:0/4194622227 shutdown_connections 2024-03-11T22:53:50.910 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:50.338+0000 7ff9deb26700 1 -- 172.21.15.153:0/4194622227 wait complete. 2024-03-11T22:53:50.911 INFO:teuthology.orchestra.run.smithi153.stdout:Generating ssh key... 2024-03-11T22:53:51.707 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:51 smithi153 ceph-mon[28962]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:53:52.271 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.334+0000 7f69b4a25700 1 Processor -- start 2024-03-11T22:53:52.271 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.334+0000 7f69b4a25700 1 -- start start 2024-03-11T22:53:52.271 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.334+0000 7f69b4a25700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69b0102770 0x7f69b0102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:52.271 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.334+0000 7f69b4a25700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b01030c0 con 0x7f69b0102770 2024-03-11T22:53:52.271 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.334+0000 7f69ae59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69b0102770 0x7f69b0102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:52.272 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.334+0000 7f69ae59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69b0102770 0x7f69b0102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:51594/0 (socket says 172.21.15.153:51594) 2024-03-11T22:53:52.272 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.334+0000 7f69ae59c700 1 -- 172.21.15.153:0/971281821 learned_addr learned my addr 172.21.15.153:0/971281821 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:53:52.272 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.335+0000 7f69ae59c700 1 -- 172.21.15.153:0/971281821 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b0103200 con 0x7f69b0102770 2024-03-11T22:53:52.272 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.335+0000 7f69ae59c700 1 --2- 172.21.15.153:0/971281821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69b0102770 0x7f69b0102b80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f69a00098e0 tx=0x7f69a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ade1564a124d8ef0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:52.272 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.335+0000 7f69ad59a700 1 -- 172.21.15.153:0/971281821 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69a000b860 con 0x7f69b0102770 2024-03-11T22:53:52.272 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.335+0000 7f69ad59a700 1 -- 172.21.15.153:0/971281821 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f69a000be50 con 0x7f69b0102770 2024-03-11T22:53:52.272 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.335+0000 7f69ad59a700 1 -- 172.21.15.153:0/971281821 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69a0010380 con 0x7f69b0102770 2024-03-11T22:53:52.272 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.335+0000 7f69b4a25700 1 -- 172.21.15.153:0/971281821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69b0102770 msgr2=0x7f69b0102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:52.272 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.335+0000 7f69b4a25700 1 --2- 172.21.15.153:0/971281821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69b0102770 0x7f69b0102b80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f69a00098e0 tx=0x7f69a0009250 comp rx=0 tx=0).stop 2024-03-11T22:53:52.272 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.336+0000 7f69b4a25700 1 -- 172.21.15.153:0/971281821 shutdown_connections 2024-03-11T22:53:52.272 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.336+0000 7f69b4a25700 1 --2- 172.21.15.153:0/971281821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69b0102770 0x7f69b0102b80 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:52.272 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.336+0000 7f69b4a25700 1 -- 172.21.15.153:0/971281821 >> 172.21.15.153:0/971281821 conn(0x7f69b00fded0 msgr2=0x7f69b0100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:53:52.272 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.336+0000 7f69b4a25700 1 -- 172.21.15.153:0/971281821 shutdown_connections 2024-03-11T22:53:52.272 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.336+0000 7f69b4a25700 1 -- 172.21.15.153:0/971281821 wait complete. 2024-03-11T22:53:52.272 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.336+0000 7f69b4a25700 1 Processor -- start 2024-03-11T22:53:52.273 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.336+0000 7f69b4a25700 1 -- start start 2024-03-11T22:53:52.273 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.336+0000 7f69b4a25700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69b0102770 0x7f69b0111560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:52.273 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.336+0000 7f69b4a25700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b0111aa0 con 0x7f69b0102770 2024-03-11T22:53:52.273 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.336+0000 7f69ae59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69b0102770 0x7f69b0111560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:52.273 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.336+0000 7f69ae59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69b0102770 0x7f69b0111560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:51610/0 (socket says 172.21.15.153:51610) 2024-03-11T22:53:52.273 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.336+0000 7f69ae59c700 1 -- 172.21.15.153:0/1547066724 learned_addr learned my addr 172.21.15.153:0/1547066724 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:53:52.273 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.337+0000 7f69ae59c700 1 -- 172.21.15.153:0/1547066724 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a00094f0 con 0x7f69b0102770 2024-03-11T22:53:52.273 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.337+0000 7f69ae59c700 1 --2- 172.21.15.153:0/1547066724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69b0102770 0x7f69b0111560 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f69a0000c00 tx=0x7f69a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:52.273 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.337+0000 7f699effd700 1 -- 172.21.15.153:0/1547066724 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69a00107c0 con 0x7f69b0102770 2024-03-11T22:53:52.273 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.337+0000 7f699effd700 1 -- 172.21.15.153:0/1547066724 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f69a0010db0 con 0x7f69b0102770 2024-03-11T22:53:52.273 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.337+0000 7f69b4a25700 1 -- 172.21.15.153:0/1547066724 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69b0111ca0 con 0x7f69b0102770 2024-03-11T22:53:52.273 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.337+0000 7f699effd700 1 -- 172.21.15.153:0/1547066724 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f69a001ad00 con 0x7f69b0102770 2024-03-11T22:53:52.273 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.337+0000 7f69b4a25700 1 -- 172.21.15.153:0/1547066724 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69b0112940 con 0x7f69b0102770 2024-03-11T22:53:52.273 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.338+0000 7f699effd700 1 -- 172.21.15.153:0/1547066724 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f69a00193c0 con 0x7f69b0102770 2024-03-11T22:53:52.274 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.338+0000 7f699effd700 1 --2- 172.21.15.153:0/1547066724 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f6998036790 0x7f6998038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:52.274 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.338+0000 7f699effd700 1 -- 172.21.15.153:0/1547066724 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f69a004a670 con 0x7f69b0102770 2024-03-11T22:53:52.274 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.338+0000 7f69add9b700 1 --2- 172.21.15.153:0/1547066724 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f6998036790 0x7f6998038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:52.274 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.338+0000 7f69b4a25700 1 -- 172.21.15.153:0/1547066724 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69b005f5c0 con 0x7f69b0102770 2024-03-11T22:53:52.274 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.339+0000 7f69add9b700 1 --2- 172.21.15.153:0/1547066724 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f6998036790 0x7f6998038c40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f69a4009990 tx=0x7f69a4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:52.274 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.343+0000 7f699effd700 1 -- 172.21.15.153:0/1547066724 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f69a00196c0 con 0x7f69b0102770 2024-03-11T22:53:52.274 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.526+0000 7f69b4a25700 1 -- 172.21.15.153:0/1547066724 --> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f69b0100190 con 0x7f6998036790 2024-03-11T22:53:52.274 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.711+0000 7f699effd700 1 -- 172.21.15.153:0/1547066724 <== mgr.14118 v2:172.21.15.153:6800/1518383615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f69b0100190 con 0x7f6998036790 2024-03-11T22:53:52.274 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.713+0000 7f69b4a25700 1 -- 172.21.15.153:0/1547066724 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f6998036790 msgr2=0x7f6998038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:52.274 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.713+0000 7f69b4a25700 1 --2- 172.21.15.153:0/1547066724 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f6998036790 0x7f6998038c40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f69a4009990 tx=0x7f69a4006e40 comp rx=0 tx=0).stop 2024-03-11T22:53:52.274 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.713+0000 7f69b4a25700 1 -- 172.21.15.153:0/1547066724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69b0102770 msgr2=0x7f69b0111560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:52.274 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.713+0000 7f69b4a25700 1 --2- 172.21.15.153:0/1547066724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69b0102770 0x7f69b0111560 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f69a0000c00 tx=0x7f69a0009250 comp rx=0 tx=0).stop 2024-03-11T22:53:52.274 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.713+0000 7f69b4a25700 1 -- 172.21.15.153:0/1547066724 shutdown_connections 2024-03-11T22:53:52.274 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.713+0000 7f69b4a25700 1 --2- 172.21.15.153:0/1547066724 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f6998036790 0x7f6998038c40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:52.275 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.713+0000 7f69b4a25700 1 --2- 172.21.15.153:0/1547066724 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69b0102770 0x7f69b0111560 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:52.275 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.713+0000 7f69b4a25700 1 -- 172.21.15.153:0/1547066724 >> 172.21.15.153:0/1547066724 conn(0x7f69b00fded0 msgr2=0x7f69b00ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:53:52.275 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.714+0000 7f69b4a25700 1 -- 172.21.15.153:0/1547066724 shutdown_connections 2024-03-11T22:53:52.275 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:51.714+0000 7f69b4a25700 1 -- 172.21.15.153:0/1547066724 wait complete. 2024-03-11T22:53:52.842 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:52 smithi153 ceph-mon[28962]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:53:52.842 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:52 smithi153 ceph-mon[28962]: Generating ssh key... 2024-03-11T22:53:52.842 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:52 smithi153 ceph-mon[28962]: from='mgr.14118 172.21.15.153:0/870689843' entity='mgr.smithi153.nyocep' 2024-03-11T22:53:52.842 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:52 smithi153 ceph-mon[28962]: from='mgr.14118 172.21.15.153:0/870689843' entity='mgr.smithi153.nyocep' 2024-03-11T22:53:53.387 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDp0ij5R7OMV1MF1G+8CFK/xpnLxZaeuNfwTyKXu1PU+hiDbbA1Iru7iy4Yfufv1sNk6XLzjZ+/oArCGtNNWb9xJUqpv4eHknSqYx1FBIYn7ibvkVwODY+oO31NuGDYrXPBJxGSKiQu+g6ysVhFOvza94GxuiSYa7DeXX0SAQL74CeTrrVgJNCi+Nacd9Gg+arTKUhkQzhOhX0EnOa7LkMulN83oSi+2Ba+F/mfDDKuUA3THOb0RSln/96G2bFV8w7kfTfPzPQzKQXMmv/MJjlOG//vPibUWD7cngYI7eGKrH+HCkg8TsFq/823h2gfmCBFkCkzesRRZHnOGwGdPnLXKOKAkSxFmkMun02BVEuUhEgimADNbrKQilPX9k0mHBnpl2zQHF0PI40nxjr1YrbuD1luVqkziRZEEz6/ahkO6flunB/auAgF6iO2FqL1NAZFY564QGdij80sN2xhgP6DfMnYxG0AFcDQCZr4P2GACZoWh9t34Bcq7gnVi1AaUtk= ceph-fa913b96-dff9-11ee-95c8-87774f69a715 2024-03-11T22:53:53.387 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.640+0000 7f7a4d8af700 1 Processor -- start 2024-03-11T22:53:53.387 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.640+0000 7f7a4d8af700 1 -- start start 2024-03-11T22:53:53.387 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.640+0000 7f7a4d8af700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a48101670 0x7f7a48101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:53.387 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.640+0000 7f7a4d8af700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a48101fc0 con 0x7f7a48101670 2024-03-11T22:53:53.387 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.641+0000 7f7a46ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a48101670 0x7f7a48101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:53.387 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.641+0000 7f7a46ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a48101670 0x7f7a48101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41048/0 (socket says 172.21.15.153:41048) 2024-03-11T22:53:53.387 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.641+0000 7f7a46ffd700 1 -- 172.21.15.153:0/1231044768 learned_addr learned my addr 172.21.15.153:0/1231044768 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:53:53.387 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.641+0000 7f7a46ffd700 1 -- 172.21.15.153:0/1231044768 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a48102100 con 0x7f7a48101670 2024-03-11T22:53:53.387 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.641+0000 7f7a46ffd700 1 --2- 172.21.15.153:0/1231044768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a48101670 0x7f7a48101a80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7a4000d0d0 tx=0x7f7a4000ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1512d4cb3a55fc98 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:53.387 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.641+0000 7f7a45ffb700 1 -- 172.21.15.153:0/1231044768 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a4000ddf0 con 0x7f7a48101670 2024-03-11T22:53:53.387 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.641+0000 7f7a45ffb700 1 -- 172.21.15.153:0/1231044768 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f7a40009610 con 0x7f7a48101670 2024-03-11T22:53:53.387 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.641+0000 7f7a45ffb700 1 -- 172.21.15.153:0/1231044768 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a400145d0 con 0x7f7a48101670 2024-03-11T22:53:53.388 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.642+0000 7f7a4d8af700 1 -- 172.21.15.153:0/1231044768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a48101670 msgr2=0x7f7a48101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:53.388 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.642+0000 7f7a4d8af700 1 --2- 172.21.15.153:0/1231044768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a48101670 0x7f7a48101a80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7a4000d0d0 tx=0x7f7a4000ba50 comp rx=0 tx=0).stop 2024-03-11T22:53:53.388 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.642+0000 7f7a4d8af700 1 -- 172.21.15.153:0/1231044768 shutdown_connections 2024-03-11T22:53:53.388 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.642+0000 7f7a4d8af700 1 --2- 172.21.15.153:0/1231044768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a48101670 0x7f7a48101a80 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:53.388 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.642+0000 7f7a4d8af700 1 -- 172.21.15.153:0/1231044768 >> 172.21.15.153:0/1231044768 conn(0x7f7a480fcdd0 msgr2=0x7f7a480ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:53:53.388 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.642+0000 7f7a4d8af700 1 -- 172.21.15.153:0/1231044768 shutdown_connections 2024-03-11T22:53:53.388 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.642+0000 7f7a4d8af700 1 -- 172.21.15.153:0/1231044768 wait complete. 2024-03-11T22:53:53.388 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.643+0000 7f7a4d8af700 1 Processor -- start 2024-03-11T22:53:53.388 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.643+0000 7f7a4d8af700 1 -- start start 2024-03-11T22:53:53.388 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.643+0000 7f7a4d8af700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a48101670 0x7f7a4810f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:53.388 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.643+0000 7f7a4d8af700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4810f8b0 con 0x7f7a48101670 2024-03-11T22:53:53.388 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.643+0000 7f7a46ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a48101670 0x7f7a4810f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:53.388 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.643+0000 7f7a46ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a48101670 0x7f7a4810f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41064/0 (socket says 172.21.15.153:41064) 2024-03-11T22:53:53.388 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.643+0000 7f7a46ffd700 1 -- 172.21.15.153:0/2206578102 learned_addr learned my addr 172.21.15.153:0/2206578102 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:53:53.388 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.643+0000 7f7a46ffd700 1 -- 172.21.15.153:0/2206578102 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a4000bcf0 con 0x7f7a48101670 2024-03-11T22:53:53.389 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.643+0000 7f7a46ffd700 1 --2- 172.21.15.153:0/2206578102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a48101670 0x7f7a4810f370 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7a4000df20 tx=0x7f7a4000ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:53.389 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.643+0000 7f7a2f7fe700 1 -- 172.21.15.153:0/2206578102 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a40009a50 con 0x7f7a48101670 2024-03-11T22:53:53.389 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.643+0000 7f7a2f7fe700 1 -- 172.21.15.153:0/2206578102 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f7a4000b2e0 con 0x7f7a48101670 2024-03-11T22:53:53.389 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.643+0000 7f7a4d8af700 1 -- 172.21.15.153:0/2206578102 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a4810fab0 con 0x7f7a48101670 2024-03-11T22:53:53.389 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.643+0000 7f7a2f7fe700 1 -- 172.21.15.153:0/2206578102 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a4001eae0 con 0x7f7a48101670 2024-03-11T22:53:53.389 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.644+0000 7f7a4d8af700 1 -- 172.21.15.153:0/2206578102 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a48110750 con 0x7f7a48101670 2024-03-11T22:53:53.389 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.644+0000 7f7a2f7fe700 1 -- 172.21.15.153:0/2206578102 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f7a400263b0 con 0x7f7a48101670 2024-03-11T22:53:53.389 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.644+0000 7f7a4d8af700 1 -- 172.21.15.153:0/2206578102 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a4805f5c0 con 0x7f7a48101670 2024-03-11T22:53:53.389 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.644+0000 7f7a2f7fe700 1 --2- 172.21.15.153:0/2206578102 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f7a30036790 0x7f7a30038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:53.389 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.644+0000 7f7a2f7fe700 1 -- 172.21.15.153:0/2206578102 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f7a4004c420 con 0x7f7a48101670 2024-03-11T22:53:53.389 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.645+0000 7f7a467fc700 1 --2- 172.21.15.153:0/2206578102 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f7a30036790 0x7f7a30038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:53.389 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.645+0000 7f7a467fc700 1 --2- 172.21.15.153:0/2206578102 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f7a30036790 0x7f7a30038c40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c009990 tx=0x7f7a3c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:53.389 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.649+0000 7f7a2f7fe700 1 -- 172.21.15.153:0/2206578102 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f7a40013930 con 0x7f7a48101670 2024-03-11T22:53:53.389 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.837+0000 7f7a4d8af700 1 -- 172.21.15.153:0/2206578102 --> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f7a48111310 con 0x7f7a30036790 2024-03-11T22:53:53.390 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.838+0000 7f7a2f7fe700 1 -- 172.21.15.153:0/2206578102 <== mgr.14118 v2:172.21.15.153:6800/1518383615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f7a48111310 con 0x7f7a30036790 2024-03-11T22:53:53.390 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.839+0000 7f7a4d8af700 1 -- 172.21.15.153:0/2206578102 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f7a30036790 msgr2=0x7f7a30038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:53.390 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.839+0000 7f7a4d8af700 1 --2- 172.21.15.153:0/2206578102 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f7a30036790 0x7f7a30038c40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c009990 tx=0x7f7a3c006e40 comp rx=0 tx=0).stop 2024-03-11T22:53:53.390 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.840+0000 7f7a4d8af700 1 -- 172.21.15.153:0/2206578102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a48101670 msgr2=0x7f7a4810f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:53.390 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.840+0000 7f7a4d8af700 1 --2- 172.21.15.153:0/2206578102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a48101670 0x7f7a4810f370 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7a4000df20 tx=0x7f7a4000ba50 comp rx=0 tx=0).stop 2024-03-11T22:53:53.390 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.840+0000 7f7a4d8af700 1 -- 172.21.15.153:0/2206578102 shutdown_connections 2024-03-11T22:53:53.390 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.840+0000 7f7a4d8af700 1 --2- 172.21.15.153:0/2206578102 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f7a30036790 0x7f7a30038c40 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:53.390 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.840+0000 7f7a4d8af700 1 --2- 172.21.15.153:0/2206578102 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7a48101670 0x7f7a4810f370 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:53.390 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.840+0000 7f7a4d8af700 1 -- 172.21.15.153:0/2206578102 >> 172.21.15.153:0/2206578102 conn(0x7f7a480fcdd0 msgr2=0x7f7a481063e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:53:53.390 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.840+0000 7f7a4d8af700 1 -- 172.21.15.153:0/2206578102 shutdown_connections 2024-03-11T22:53:53.390 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:52.840+0000 7f7a4d8af700 1 -- 172.21.15.153:0/2206578102 wait complete. 2024-03-11T22:53:53.390 INFO:teuthology.orchestra.run.smithi153.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-11T22:53:53.390 INFO:teuthology.orchestra.run.smithi153.stdout:Adding key to root@localhost authorized_keys... 2024-03-11T22:53:53.390 INFO:teuthology.orchestra.run.smithi153.stdout:Adding host smithi153... 2024-03-11T22:53:53.995 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:53 smithi153 ceph-mon[28962]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:53:55.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:55 smithi153 ceph-mon[28962]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi153", "addr": "172.21.15.153", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:53:56.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:56 smithi153 ceph-mon[28962]: Deploying cephadm binary to smithi153 2024-03-11T22:53:56.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:56 smithi153 ceph-mon[28962]: from='mgr.14118 172.21.15.153:0/870689843' entity='mgr.smithi153.nyocep' 2024-03-11T22:53:56.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:56 smithi153 ceph-mon[28962]: from='mgr.14118 172.21.15.153:0/870689843' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:53:56.554 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Added host 'smithi153' with addr '172.21.15.153' 2024-03-11T22:53:56.554 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.794+0000 7f177df65700 1 Processor -- start 2024-03-11T22:53:56.554 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.794+0000 7f177df65700 1 -- start start 2024-03-11T22:53:56.554 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.794+0000 7f177df65700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1778078d00 0x7f1778077210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:56.554 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.794+0000 7f177df65700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1778079110 con 0x7f1778078d00 2024-03-11T22:53:56.554 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.794+0000 7f17777fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1778078d00 0x7f1778077210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:56.554 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.794+0000 7f17777fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1778078d00 0x7f1778077210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41072/0 (socket says 172.21.15.153:41072) 2024-03-11T22:53:56.554 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.794+0000 7f17777fe700 1 -- 172.21.15.153:0/3197923289 learned_addr learned my addr 172.21.15.153:0/3197923289 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:53:56.554 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.794+0000 7f17777fe700 1 -- 172.21.15.153:0/3197923289 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1778077750 con 0x7f1778078d00 2024-03-11T22:53:56.554 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.794+0000 7f17777fe700 1 --2- 172.21.15.153:0/3197923289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1778078d00 0x7f1778077210 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f17680098e0 tx=0x7f1768009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb1679aa94c47987 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:56.554 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.795+0000 7f17767fc700 1 -- 172.21.15.153:0/3197923289 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f176800b860 con 0x7f1778078d00 2024-03-11T22:53:56.554 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.795+0000 7f17767fc700 1 -- 172.21.15.153:0/3197923289 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f176800be50 con 0x7f1778078d00 2024-03-11T22:53:56.555 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.795+0000 7f17767fc700 1 -- 172.21.15.153:0/3197923289 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1768010380 con 0x7f1778078d00 2024-03-11T22:53:56.555 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.795+0000 7f177df65700 1 -- 172.21.15.153:0/3197923289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1778078d00 msgr2=0x7f1778077210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:56.555 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.795+0000 7f177df65700 1 --2- 172.21.15.153:0/3197923289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1778078d00 0x7f1778077210 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f17680098e0 tx=0x7f1768009250 comp rx=0 tx=0).stop 2024-03-11T22:53:56.555 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.795+0000 7f177df65700 1 -- 172.21.15.153:0/3197923289 shutdown_connections 2024-03-11T22:53:56.555 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.795+0000 7f177df65700 1 --2- 172.21.15.153:0/3197923289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1778078d00 0x7f1778077210 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:56.555 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.795+0000 7f177df65700 1 -- 172.21.15.153:0/3197923289 >> 172.21.15.153:0/3197923289 conn(0x7f17780fded0 msgr2=0x7f1778100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:53:56.555 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.795+0000 7f177df65700 1 -- 172.21.15.153:0/3197923289 shutdown_connections 2024-03-11T22:53:56.555 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.795+0000 7f177df65700 1 -- 172.21.15.153:0/3197923289 wait complete. 2024-03-11T22:53:56.555 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.796+0000 7f177df65700 1 Processor -- start 2024-03-11T22:53:56.555 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.796+0000 7f177df65700 1 -- start start 2024-03-11T22:53:56.555 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.796+0000 7f177df65700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1778078d00 0x7f17781137e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:56.555 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.796+0000 7f177df65700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1778113d20 con 0x7f1778078d00 2024-03-11T22:53:56.555 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.796+0000 7f17777fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1778078d00 0x7f17781137e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:56.555 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.796+0000 7f17777fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1778078d00 0x7f17781137e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41078/0 (socket says 172.21.15.153:41078) 2024-03-11T22:53:56.555 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.796+0000 7f17777fe700 1 -- 172.21.15.153:0/3353018363 learned_addr learned my addr 172.21.15.153:0/3353018363 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:53:56.556 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.797+0000 7f17777fe700 1 -- 172.21.15.153:0/3353018363 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17680094f0 con 0x7f1778078d00 2024-03-11T22:53:56.556 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.797+0000 7f17777fe700 1 --2- 172.21.15.153:0/3353018363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1778078d00 0x7f17781137e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1768003050 tx=0x7f1768009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:56.556 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.797+0000 7f175ffff700 1 -- 172.21.15.153:0/3353018363 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1768010760 con 0x7f1778078d00 2024-03-11T22:53:56.556 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.797+0000 7f175ffff700 1 -- 172.21.15.153:0/3353018363 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f1768010d50 con 0x7f1778078d00 2024-03-11T22:53:56.556 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.797+0000 7f177df65700 1 -- 172.21.15.153:0/3353018363 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1778113f20 con 0x7f1778078d00 2024-03-11T22:53:56.556 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.797+0000 7f175ffff700 1 -- 172.21.15.153:0/3353018363 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f176801ae20 con 0x7f1778078d00 2024-03-11T22:53:56.556 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.797+0000 7f177df65700 1 -- 172.21.15.153:0/3353018363 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1778114bc0 con 0x7f1778078d00 2024-03-11T22:53:56.556 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.798+0000 7f175ffff700 1 -- 172.21.15.153:0/3353018363 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f176802e440 con 0x7f1778078d00 2024-03-11T22:53:56.556 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.798+0000 7f175ffff700 1 --2- 172.21.15.153:0/3353018363 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f1760036770 0x7f1760038c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:56.556 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.798+0000 7f175ffff700 1 -- 172.21.15.153:0/3353018363 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f176804b4f0 con 0x7f1778078d00 2024-03-11T22:53:56.556 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.798+0000 7f1776ffd700 1 --2- 172.21.15.153:0/3353018363 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f1760036770 0x7f1760038c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:56.556 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.798+0000 7f177df65700 1 -- 172.21.15.153:0/3353018363 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17781143b0 con 0x7f1778078d00 2024-03-11T22:53:56.556 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.798+0000 7f1776ffd700 1 --2- 172.21.15.153:0/3353018363 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f1760036770 0x7f1760038c20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f176c009990 tx=0x7f176c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:56.556 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.803+0000 7f175ffff700 1 -- 172.21.15.153:0/3353018363 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f1768018940 con 0x7f1778078d00 2024-03-11T22:53:56.556 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:53.990+0000 7f177df65700 1 -- 172.21.15.153:0/3353018363 --> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi153", "addr": "172.21.15.153", "target": ["mon-mgr", ""]}) v1 -- 0x7f1778100240 con 0x7f1760036770 2024-03-11T22:53:56.557 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.017+0000 7f175ffff700 1 -- 172.21.15.153:0/3353018363 <== mgr.14118 v2:172.21.15.153:6800/1518383615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f1778100240 con 0x7f1760036770 2024-03-11T22:53:56.557 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.019+0000 7f177df65700 1 -- 172.21.15.153:0/3353018363 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f1760036770 msgr2=0x7f1760038c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:56.557 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.019+0000 7f177df65700 1 --2- 172.21.15.153:0/3353018363 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f1760036770 0x7f1760038c20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f176c009990 tx=0x7f176c006e40 comp rx=0 tx=0).stop 2024-03-11T22:53:56.557 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.019+0000 7f177df65700 1 -- 172.21.15.153:0/3353018363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1778078d00 msgr2=0x7f17781137e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:56.557 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.019+0000 7f177df65700 1 --2- 172.21.15.153:0/3353018363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1778078d00 0x7f17781137e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1768003050 tx=0x7f1768009250 comp rx=0 tx=0).stop 2024-03-11T22:53:56.557 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.019+0000 7f177df65700 1 -- 172.21.15.153:0/3353018363 shutdown_connections 2024-03-11T22:53:56.557 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.019+0000 7f177df65700 1 --2- 172.21.15.153:0/3353018363 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f1760036770 0x7f1760038c20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:56.557 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.019+0000 7f177df65700 1 --2- 172.21.15.153:0/3353018363 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1778078d00 0x7f17781137e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:56.557 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.019+0000 7f177df65700 1 -- 172.21.15.153:0/3353018363 >> 172.21.15.153:0/3353018363 conn(0x7f17780fded0 msgr2=0x7f17780ff620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:53:56.557 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.020+0000 7f177df65700 1 -- 172.21.15.153:0/3353018363 shutdown_connections 2024-03-11T22:53:56.557 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.020+0000 7f177df65700 1 -- 172.21.15.153:0/3353018363 wait complete. 2024-03-11T22:53:56.557 INFO:teuthology.orchestra.run.smithi153.stdout:Deploying mon service with default placement... 2024-03-11T22:53:57.504 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:57 smithi153 ceph-mon[28962]: Added host smithi153 2024-03-11T22:53:57.505 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:57 smithi153 ceph-mon[28962]: from='mgr.14118 172.21.15.153:0/870689843' entity='mgr.smithi153.nyocep' 2024-03-11T22:53:57.880 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-11T22:53:57.880 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.952+0000 7fda8a2f8700 1 Processor -- start 2024-03-11T22:53:57.880 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.952+0000 7fda8a2f8700 1 -- start start 2024-03-11T22:53:57.880 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.952+0000 7fda8a2f8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda84102770 0x7fda84102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:57.880 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.952+0000 7fda8a2f8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda841030c0 con 0x7fda84102770 2024-03-11T22:53:57.880 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.953+0000 7fda83fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda84102770 0x7fda84102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:57.880 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.953+0000 7fda83fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda84102770 0x7fda84102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41090/0 (socket says 172.21.15.153:41090) 2024-03-11T22:53:57.880 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.953+0000 7fda83fff700 1 -- 172.21.15.153:0/3432770216 learned_addr learned my addr 172.21.15.153:0/3432770216 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:53:57.880 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.953+0000 7fda83fff700 1 -- 172.21.15.153:0/3432770216 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda84103200 con 0x7fda84102770 2024-03-11T22:53:57.880 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.954+0000 7fda83fff700 1 --2- 172.21.15.153:0/3432770216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda84102770 0x7fda84102b80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fda6c0098e0 tx=0x7fda6c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba0b7a2c26d8e264 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:57.881 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.954+0000 7fda82ffd700 1 -- 172.21.15.153:0/3432770216 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda6c00b860 con 0x7fda84102770 2024-03-11T22:53:57.881 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.954+0000 7fda82ffd700 1 -- 172.21.15.153:0/3432770216 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fda6c00be50 con 0x7fda84102770 2024-03-11T22:53:57.881 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.954+0000 7fda82ffd700 1 -- 172.21.15.153:0/3432770216 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda6c010380 con 0x7fda84102770 2024-03-11T22:53:57.881 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.955+0000 7fda8a2f8700 1 -- 172.21.15.153:0/3432770216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda84102770 msgr2=0x7fda84102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:57.881 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.955+0000 7fda8a2f8700 1 --2- 172.21.15.153:0/3432770216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda84102770 0x7fda84102b80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fda6c0098e0 tx=0x7fda6c009250 comp rx=0 tx=0).stop 2024-03-11T22:53:57.881 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.955+0000 7fda8a2f8700 1 -- 172.21.15.153:0/3432770216 shutdown_connections 2024-03-11T22:53:57.881 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.955+0000 7fda8a2f8700 1 --2- 172.21.15.153:0/3432770216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda84102770 0x7fda84102b80 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:57.881 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.955+0000 7fda8a2f8700 1 -- 172.21.15.153:0/3432770216 >> 172.21.15.153:0/3432770216 conn(0x7fda840fded0 msgr2=0x7fda84100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:53:57.881 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.955+0000 7fda8a2f8700 1 -- 172.21.15.153:0/3432770216 shutdown_connections 2024-03-11T22:53:57.881 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.955+0000 7fda8a2f8700 1 -- 172.21.15.153:0/3432770216 wait complete. 2024-03-11T22:53:57.881 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.956+0000 7fda8a2f8700 1 Processor -- start 2024-03-11T22:53:57.881 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.956+0000 7fda8a2f8700 1 -- start start 2024-03-11T22:53:57.881 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.956+0000 7fda8a2f8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda84102770 0x7fda84111500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:57.881 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.956+0000 7fda8a2f8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda84111a40 con 0x7fda84102770 2024-03-11T22:53:57.881 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.956+0000 7fda83fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda84102770 0x7fda84111500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:57.882 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.956+0000 7fda83fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda84102770 0x7fda84111500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41100/0 (socket says 172.21.15.153:41100) 2024-03-11T22:53:57.882 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.956+0000 7fda83fff700 1 -- 172.21.15.153:0/1702861227 learned_addr learned my addr 172.21.15.153:0/1702861227 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:53:57.882 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.956+0000 7fda83fff700 1 -- 172.21.15.153:0/1702861227 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda6c0094f0 con 0x7fda84102770 2024-03-11T22:53:57.882 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.956+0000 7fda83fff700 1 --2- 172.21.15.153:0/1702861227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda84102770 0x7fda84111500 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fda6c00b440 tx=0x7fda6c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:57.882 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.957+0000 7fda80ff9700 1 -- 172.21.15.153:0/1702861227 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda6c010760 con 0x7fda84102770 2024-03-11T22:53:57.882 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.957+0000 7fda80ff9700 1 -- 172.21.15.153:0/1702861227 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fda6c010d50 con 0x7fda84102770 2024-03-11T22:53:57.882 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.957+0000 7fda80ff9700 1 -- 172.21.15.153:0/1702861227 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda6c024d80 con 0x7fda84102770 2024-03-11T22:53:57.882 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.957+0000 7fda8a2f8700 1 -- 172.21.15.153:0/1702861227 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda84111c40 con 0x7fda84102770 2024-03-11T22:53:57.882 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.957+0000 7fda8a2f8700 1 -- 172.21.15.153:0/1702861227 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda841128e0 con 0x7fda84102770 2024-03-11T22:53:57.882 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.958+0000 7fda80ff9700 1 -- 172.21.15.153:0/1702861227 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fda6c01a430 con 0x7fda84102770 2024-03-11T22:53:57.882 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.958+0000 7fda8a2f8700 1 -- 172.21.15.153:0/1702861227 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda8405f5c0 con 0x7fda84102770 2024-03-11T22:53:57.882 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.958+0000 7fda80ff9700 1 --2- 172.21.15.153:0/1702861227 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fda70036740 0x7fda70038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:57.882 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.958+0000 7fda80ff9700 1 -- 172.21.15.153:0/1702861227 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fda6c04a8b0 con 0x7fda84102770 2024-03-11T22:53:57.883 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.958+0000 7fda837fe700 1 --2- 172.21.15.153:0/1702861227 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fda70036740 0x7fda70038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:57.883 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.959+0000 7fda837fe700 1 --2- 172.21.15.153:0/1702861227 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fda70036740 0x7fda70038bf0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fda74009990 tx=0x7fda74006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:57.883 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:56.963+0000 7fda80ff9700 1 -- 172.21.15.153:0/1702861227 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7fda6c018940 con 0x7fda84102770 2024-03-11T22:53:57.883 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:57.163+0000 7fda8a2f8700 1 -- 172.21.15.153:0/1702861227 --> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7fda840ff380 con 0x7fda70036740 2024-03-11T22:53:57.883 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:57.167+0000 7fda80ff9700 1 -- 172.21.15.153:0/1702861227 <== mgr.14118 v2:172.21.15.153:6800/1518383615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fda840ff380 con 0x7fda70036740 2024-03-11T22:53:57.883 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:57.169+0000 7fda8a2f8700 1 -- 172.21.15.153:0/1702861227 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fda70036740 msgr2=0x7fda70038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:57.883 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:57.169+0000 7fda8a2f8700 1 --2- 172.21.15.153:0/1702861227 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fda70036740 0x7fda70038bf0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fda74009990 tx=0x7fda74006e40 comp rx=0 tx=0).stop 2024-03-11T22:53:57.883 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:57.169+0000 7fda8a2f8700 1 -- 172.21.15.153:0/1702861227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda84102770 msgr2=0x7fda84111500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:57.883 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:57.169+0000 7fda8a2f8700 1 --2- 172.21.15.153:0/1702861227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda84102770 0x7fda84111500 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fda6c00b440 tx=0x7fda6c009250 comp rx=0 tx=0).stop 2024-03-11T22:53:57.883 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:57.169+0000 7fda8a2f8700 1 -- 172.21.15.153:0/1702861227 shutdown_connections 2024-03-11T22:53:57.883 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:57.169+0000 7fda8a2f8700 1 --2- 172.21.15.153:0/1702861227 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fda70036740 0x7fda70038bf0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:57.883 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:57.169+0000 7fda8a2f8700 1 --2- 172.21.15.153:0/1702861227 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda84102770 0x7fda84111500 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:57.883 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:57.169+0000 7fda8a2f8700 1 -- 172.21.15.153:0/1702861227 >> 172.21.15.153:0/1702861227 conn(0x7fda840fded0 msgr2=0x7fda840fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:53:57.883 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:57.170+0000 7fda8a2f8700 1 -- 172.21.15.153:0/1702861227 shutdown_connections 2024-03-11T22:53:57.883 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:57.170+0000 7fda8a2f8700 1 -- 172.21.15.153:0/1702861227 wait complete. 2024-03-11T22:53:57.884 INFO:teuthology.orchestra.run.smithi153.stdout:Deploying mgr service with default placement... 2024-03-11T22:53:58.428 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:58 smithi153 ceph-mon[28962]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:53:58.428 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:53:58 smithi153 ceph-mon[28962]: Saving service mon spec with placement count:5 2024-03-11T22:53:59.682 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-11T22:53:59.682 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.794+0000 7fc886e4b700 1 Processor -- start 2024-03-11T22:53:59.682 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.794+0000 7fc886e4b700 1 -- start start 2024-03-11T22:53:59.682 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.794+0000 7fc886e4b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc880078d00 0x7fc880077210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:59.683 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.794+0000 7fc886e4b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc880079110 con 0x7fc880078d00 2024-03-11T22:53:59.683 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.794+0000 7fc884be7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc880078d00 0x7fc880077210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:59.683 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.794+0000 7fc884be7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc880078d00 0x7fc880077210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41112/0 (socket says 172.21.15.153:41112) 2024-03-11T22:53:59.683 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.794+0000 7fc884be7700 1 -- 172.21.15.153:0/234127143 learned_addr learned my addr 172.21.15.153:0/234127143 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:53:59.683 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.794+0000 7fc884be7700 1 -- 172.21.15.153:0/234127143 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc880077750 con 0x7fc880078d00 2024-03-11T22:53:59.683 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.795+0000 7fc884be7700 1 --2- 172.21.15.153:0/234127143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc880078d00 0x7fc880077210 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc8740098e0 tx=0x7fc874009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd3843097c940c87 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:59.688 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.795+0000 7fc87f7fe700 1 -- 172.21.15.153:0/234127143 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc87400b860 con 0x7fc880078d00 2024-03-11T22:53:59.688 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.795+0000 7fc87f7fe700 1 -- 172.21.15.153:0/234127143 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fc87400be50 con 0x7fc880078d00 2024-03-11T22:53:59.688 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.795+0000 7fc87f7fe700 1 -- 172.21.15.153:0/234127143 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc874010380 con 0x7fc880078d00 2024-03-11T22:53:59.688 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.796+0000 7fc886e4b700 1 -- 172.21.15.153:0/234127143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc880078d00 msgr2=0x7fc880077210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:59.688 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.796+0000 7fc886e4b700 1 --2- 172.21.15.153:0/234127143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc880078d00 0x7fc880077210 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc8740098e0 tx=0x7fc874009250 comp rx=0 tx=0).stop 2024-03-11T22:53:59.688 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.796+0000 7fc886e4b700 1 -- 172.21.15.153:0/234127143 shutdown_connections 2024-03-11T22:53:59.688 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.796+0000 7fc886e4b700 1 --2- 172.21.15.153:0/234127143 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc880078d00 0x7fc880077210 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:59.688 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.796+0000 7fc886e4b700 1 -- 172.21.15.153:0/234127143 >> 172.21.15.153:0/234127143 conn(0x7fc8800fdf00 msgr2=0x7fc880100310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:53:59.688 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.796+0000 7fc886e4b700 1 -- 172.21.15.153:0/234127143 shutdown_connections 2024-03-11T22:53:59.688 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.796+0000 7fc886e4b700 1 -- 172.21.15.153:0/234127143 wait complete. 2024-03-11T22:53:59.688 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.797+0000 7fc886e4b700 1 Processor -- start 2024-03-11T22:53:59.688 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.797+0000 7fc886e4b700 1 -- start start 2024-03-11T22:53:59.689 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.797+0000 7fc886e4b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc880078d00 0x7fc8801137b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:59.689 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.797+0000 7fc886e4b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc880113cf0 con 0x7fc880078d00 2024-03-11T22:53:59.689 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.797+0000 7fc884be7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc880078d00 0x7fc8801137b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:59.689 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.797+0000 7fc884be7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc880078d00 0x7fc8801137b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41120/0 (socket says 172.21.15.153:41120) 2024-03-11T22:53:59.689 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.797+0000 7fc884be7700 1 -- 172.21.15.153:0/4267131157 learned_addr learned my addr 172.21.15.153:0/4267131157 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:53:59.689 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.797+0000 7fc884be7700 1 -- 172.21.15.153:0/4267131157 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8740094f0 con 0x7fc880078d00 2024-03-11T22:53:59.689 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.798+0000 7fc884be7700 1 --2- 172.21.15.153:0/4267131157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc880078d00 0x7fc8801137b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc874006b00 tx=0x7fc874009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:59.689 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.798+0000 7fc87d7fa700 1 -- 172.21.15.153:0/4267131157 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8740107c0 con 0x7fc880078d00 2024-03-11T22:53:59.689 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.798+0000 7fc87d7fa700 1 -- 172.21.15.153:0/4267131157 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fc874010db0 con 0x7fc880078d00 2024-03-11T22:53:59.689 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.798+0000 7fc886e4b700 1 -- 172.21.15.153:0/4267131157 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc880113ef0 con 0x7fc880078d00 2024-03-11T22:53:59.689 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.798+0000 7fc87d7fa700 1 -- 172.21.15.153:0/4267131157 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc874024cd0 con 0x7fc880078d00 2024-03-11T22:53:59.689 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.798+0000 7fc886e4b700 1 -- 172.21.15.153:0/4267131157 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc880114b90 con 0x7fc880078d00 2024-03-11T22:53:59.689 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.799+0000 7fc87d7fa700 1 -- 172.21.15.153:0/4267131157 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fc87401a430 con 0x7fc880078d00 2024-03-11T22:53:59.689 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.799+0000 7fc886e4b700 1 -- 172.21.15.153:0/4267131157 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc88005f5c0 con 0x7fc880078d00 2024-03-11T22:53:59.689 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.799+0000 7fc87d7fa700 1 --2- 172.21.15.153:0/4267131157 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fc8700367e0 0x7fc870038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:53:59.690 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.799+0000 7fc87d7fa700 1 -- 172.21.15.153:0/4267131157 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fc87404a520 con 0x7fc880078d00 2024-03-11T22:53:59.690 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.799+0000 7fc87ffff700 1 --2- 172.21.15.153:0/4267131157 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fc8700367e0 0x7fc870038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:53:59.690 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.799+0000 7fc87ffff700 1 --2- 172.21.15.153:0/4267131157 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fc8700367e0 0x7fc870038c90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc86c009990 tx=0x7fc86c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:53:59.690 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:58.804+0000 7fc87d7fa700 1 -- 172.21.15.153:0/4267131157 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7fc87401a6e0 con 0x7fc880078d00 2024-03-11T22:53:59.690 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:59.000+0000 7fc886e4b700 1 -- 172.21.15.153:0/4267131157 --> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fc880100210 con 0x7fc8700367e0 2024-03-11T22:53:59.690 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:59.004+0000 7fc87d7fa700 1 -- 172.21.15.153:0/4267131157 <== mgr.14118 v2:172.21.15.153:6800/1518383615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fc880100210 con 0x7fc8700367e0 2024-03-11T22:53:59.690 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:59.006+0000 7fc886e4b700 1 -- 172.21.15.153:0/4267131157 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fc8700367e0 msgr2=0x7fc870038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:59.690 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:59.006+0000 7fc886e4b700 1 --2- 172.21.15.153:0/4267131157 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fc8700367e0 0x7fc870038c90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc86c009990 tx=0x7fc86c006e40 comp rx=0 tx=0).stop 2024-03-11T22:53:59.690 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:59.006+0000 7fc886e4b700 1 -- 172.21.15.153:0/4267131157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc880078d00 msgr2=0x7fc8801137b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:53:59.690 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:59.006+0000 7fc886e4b700 1 --2- 172.21.15.153:0/4267131157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc880078d00 0x7fc8801137b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc874006b00 tx=0x7fc874009250 comp rx=0 tx=0).stop 2024-03-11T22:53:59.690 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:59.006+0000 7fc886e4b700 1 -- 172.21.15.153:0/4267131157 shutdown_connections 2024-03-11T22:53:59.690 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:59.006+0000 7fc886e4b700 1 --2- 172.21.15.153:0/4267131157 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fc8700367e0 0x7fc870038c90 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:59.690 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:59.006+0000 7fc886e4b700 1 --2- 172.21.15.153:0/4267131157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc880078d00 0x7fc8801137b0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:53:59.691 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:59.006+0000 7fc886e4b700 1 -- 172.21.15.153:0/4267131157 >> 172.21.15.153:0/4267131157 conn(0x7fc8800fdf00 msgr2=0x7fc8800ff5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:53:59.691 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:59.006+0000 7fc886e4b700 1 -- 172.21.15.153:0/4267131157 shutdown_connections 2024-03-11T22:53:59.691 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:53:59.006+0000 7fc886e4b700 1 -- 172.21.15.153:0/4267131157 wait complete. 2024-03-11T22:53:59.691 INFO:teuthology.orchestra.run.smithi153.stdout:Deploying crash service with default placement... 2024-03-11T22:54:00.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:00 smithi153 ceph-mon[28962]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:54:00.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:00 smithi153 ceph-mon[28962]: Saving service mgr spec with placement count:2 2024-03-11T22:54:00.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:00 smithi153 ceph-mon[28962]: from='mgr.14118 172.21.15.153:0/870689843' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:00.032 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:00 smithi153 ceph-mon[28962]: from='mgr.14118 172.21.15.153:0/870689843' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:00.841 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-03-11T22:54:00.841 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.037+0000 7f6c81054700 1 Processor -- start 2024-03-11T22:54:00.842 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.037+0000 7f6c81054700 1 -- start start 2024-03-11T22:54:00.842 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.038+0000 7f6c81054700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c7c101670 0x7f6c7c101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:00.842 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.038+0000 7f6c81054700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c101fc0 con 0x7f6c7c101670 2024-03-11T22:54:00.842 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.038+0000 7f6c7ad9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c7c101670 0x7f6c7c101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:00.842 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.038+0000 7f6c7ad9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c7c101670 0x7f6c7c101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41128/0 (socket says 172.21.15.153:41128) 2024-03-11T22:54:00.842 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.038+0000 7f6c7ad9d700 1 -- 172.21.15.153:0/1509595913 learned_addr learned my addr 172.21.15.153:0/1509595913 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:54:00.842 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.038+0000 7f6c7ad9d700 1 -- 172.21.15.153:0/1509595913 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c7c102100 con 0x7f6c7c101670 2024-03-11T22:54:00.842 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.038+0000 7f6c7ad9d700 1 --2- 172.21.15.153:0/1509595913 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c7c101670 0x7f6c7c101a80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6c700098e0 tx=0x7f6c70009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f335ef0be6ee0914 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:00.842 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.038+0000 7f6c79d9b700 1 -- 172.21.15.153:0/1509595913 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c7000b860 con 0x7f6c7c101670 2024-03-11T22:54:00.842 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.038+0000 7f6c79d9b700 1 -- 172.21.15.153:0/1509595913 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6c7000be90 con 0x7f6c7c101670 2024-03-11T22:54:00.842 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.038+0000 7f6c79d9b700 1 -- 172.21.15.153:0/1509595913 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c700103c0 con 0x7f6c7c101670 2024-03-11T22:54:00.842 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.039+0000 7f6c81054700 1 -- 172.21.15.153:0/1509595913 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c7c101670 msgr2=0x7f6c7c101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:00.843 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.039+0000 7f6c81054700 1 --2- 172.21.15.153:0/1509595913 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c7c101670 0x7f6c7c101a80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6c700098e0 tx=0x7f6c70009250 comp rx=0 tx=0).stop 2024-03-11T22:54:00.843 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.039+0000 7f6c81054700 1 -- 172.21.15.153:0/1509595913 shutdown_connections 2024-03-11T22:54:00.843 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.039+0000 7f6c81054700 1 --2- 172.21.15.153:0/1509595913 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c7c101670 0x7f6c7c101a80 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:00.843 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.039+0000 7f6c81054700 1 -- 172.21.15.153:0/1509595913 >> 172.21.15.153:0/1509595913 conn(0x7f6c7c0fcdd0 msgr2=0x7f6c7c0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:00.843 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.039+0000 7f6c81054700 1 -- 172.21.15.153:0/1509595913 shutdown_connections 2024-03-11T22:54:00.843 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.039+0000 7f6c81054700 1 -- 172.21.15.153:0/1509595913 wait complete. 2024-03-11T22:54:00.843 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.039+0000 7f6c81054700 1 Processor -- start 2024-03-11T22:54:00.843 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.040+0000 7f6c81054700 1 -- start start 2024-03-11T22:54:00.843 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.040+0000 7f6c81054700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c7c101670 0x7f6c7c114580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:00.843 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.040+0000 7f6c81054700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c076bf0 con 0x7f6c7c101670 2024-03-11T22:54:00.843 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.040+0000 7f6c7ad9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c7c101670 0x7f6c7c114580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:00.843 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.040+0000 7f6c7ad9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c7c101670 0x7f6c7c114580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41140/0 (socket says 172.21.15.153:41140) 2024-03-11T22:54:00.843 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.040+0000 7f6c7ad9d700 1 -- 172.21.15.153:0/1964972889 learned_addr learned my addr 172.21.15.153:0/1964972889 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:54:00.843 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.040+0000 7f6c7ad9d700 1 -- 172.21.15.153:0/1964972889 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c700094f0 con 0x7f6c7c101670 2024-03-11T22:54:00.844 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.040+0000 7f6c7ad9d700 1 --2- 172.21.15.153:0/1964972889 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c7c101670 0x7f6c7c114580 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6c70011650 tx=0x7f6c70009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:00.844 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.040+0000 7f6c637fe700 1 -- 172.21.15.153:0/1964972889 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c70010890 con 0x7f6c7c101670 2024-03-11T22:54:00.844 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.040+0000 7f6c637fe700 1 -- 172.21.15.153:0/1964972889 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6c70010ec0 con 0x7f6c7c101670 2024-03-11T22:54:00.844 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.040+0000 7f6c637fe700 1 -- 172.21.15.153:0/1964972889 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c7001a430 con 0x7f6c7c101670 2024-03-11T22:54:00.844 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.040+0000 7f6c81054700 1 -- 172.21.15.153:0/1964972889 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c7c076df0 con 0x7f6c7c101670 2024-03-11T22:54:00.844 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.040+0000 7f6c81054700 1 -- 172.21.15.153:0/1964972889 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c7c114c80 con 0x7f6c7c101670 2024-03-11T22:54:00.844 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.041+0000 7f6c637fe700 1 -- 172.21.15.153:0/1964972889 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f6c7001a6b0 con 0x7f6c7c101670 2024-03-11T22:54:00.844 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.041+0000 7f6c617fa700 1 -- 172.21.15.153:0/1964972889 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c5c0052b0 con 0x7f6c7c101670 2024-03-11T22:54:00.844 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.041+0000 7f6c637fe700 1 --2- 172.21.15.153:0/1964972889 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f6c64036790 0x7f6c64038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:00.844 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.041+0000 7f6c637fe700 1 -- 172.21.15.153:0/1964972889 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f6c7004a690 con 0x7f6c7c101670 2024-03-11T22:54:00.844 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.041+0000 7f6c7a59c700 1 --2- 172.21.15.153:0/1964972889 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f6c64036790 0x7f6c64038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:00.844 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.042+0000 7f6c7a59c700 1 --2- 172.21.15.153:0/1964972889 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f6c64036790 0x7f6c64038c40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c009990 tx=0x7f6c6c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:00.844 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.046+0000 7f6c637fe700 1 -- 172.21.15.153:0/1964972889 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f6c70018db0 con 0x7f6c7c101670 2024-03-11T22:54:00.844 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.249+0000 7f6c617fa700 1 -- 172.21.15.153:0/1964972889 --> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f6c5c000bc0 con 0x7f6c64036790 2024-03-11T22:54:00.845 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.256+0000 7f6c637fe700 1 -- 172.21.15.153:0/1964972889 <== mgr.14118 v2:172.21.15.153:6800/1518383615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f6c5c000bc0 con 0x7f6c64036790 2024-03-11T22:54:00.845 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.258+0000 7f6c617fa700 1 -- 172.21.15.153:0/1964972889 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f6c64036790 msgr2=0x7f6c64038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:00.845 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.258+0000 7f6c617fa700 1 --2- 172.21.15.153:0/1964972889 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f6c64036790 0x7f6c64038c40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c009990 tx=0x7f6c6c006e40 comp rx=0 tx=0).stop 2024-03-11T22:54:00.845 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.258+0000 7f6c617fa700 1 -- 172.21.15.153:0/1964972889 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c7c101670 msgr2=0x7f6c7c114580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:00.845 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.258+0000 7f6c617fa700 1 --2- 172.21.15.153:0/1964972889 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c7c101670 0x7f6c7c114580 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6c70011650 tx=0x7f6c70009250 comp rx=0 tx=0).stop 2024-03-11T22:54:00.845 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.258+0000 7f6c617fa700 1 -- 172.21.15.153:0/1964972889 shutdown_connections 2024-03-11T22:54:00.845 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.258+0000 7f6c617fa700 1 --2- 172.21.15.153:0/1964972889 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f6c64036790 0x7f6c64038c40 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:00.845 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.258+0000 7f6c617fa700 1 --2- 172.21.15.153:0/1964972889 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c7c101670 0x7f6c7c114580 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:00.845 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.258+0000 7f6c617fa700 1 -- 172.21.15.153:0/1964972889 >> 172.21.15.153:0/1964972889 conn(0x7f6c7c0fcdd0 msgr2=0x7f6c7c0fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:00.845 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.258+0000 7f6c617fa700 1 -- 172.21.15.153:0/1964972889 shutdown_connections 2024-03-11T22:54:00.845 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:00.258+0000 7f6c617fa700 1 -- 172.21.15.153:0/1964972889 wait complete. 2024-03-11T22:54:00.845 INFO:teuthology.orchestra.run.smithi153.stdout:Deploying prometheus service with default placement... 2024-03-11T22:54:01.438 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:01 smithi153 ceph-mon[28962]: from='mgr.14118 172.21.15.153:0/870689843' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:01.438 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:01 smithi153 ceph-mon[28962]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:54:01.438 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:01 smithi153 ceph-mon[28962]: Saving service crash spec with placement * 2024-03-11T22:54:01.439 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:01 smithi153 ceph-mon[28962]: from='mgr.14118 172.21.15.153:0/870689843' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:02.141 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-03-11T22:54:02.141 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.186+0000 7fbafb50b700 1 Processor -- start 2024-03-11T22:54:02.142 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.186+0000 7fbafb50b700 1 -- start start 2024-03-11T22:54:02.142 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.186+0000 7fbafb50b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbaf4103e40 0x7fbaf4104250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:02.142 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.186+0000 7fbafb50b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf4104790 con 0x7fbaf4103e40 2024-03-11T22:54:02.142 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.186+0000 7fbaf92a7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbaf4103e40 0x7fbaf4104250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:02.142 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.187+0000 7fbaf92a7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbaf4103e40 0x7fbaf4104250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41144/0 (socket says 172.21.15.153:41144) 2024-03-11T22:54:02.142 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.187+0000 7fbaf92a7700 1 -- 172.21.15.153:0/1972726557 learned_addr learned my addr 172.21.15.153:0/1972726557 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:54:02.142 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.187+0000 7fbaf92a7700 1 -- 172.21.15.153:0/1972726557 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf41048d0 con 0x7fbaf4103e40 2024-03-11T22:54:02.142 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.187+0000 7fbaf92a7700 1 --2- 172.21.15.153:0/1972726557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbaf4103e40 0x7fbaf4104250 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbaf00098e0 tx=0x7fbaf0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feb4dfc62ecf1a48 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:02.142 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.187+0000 7fbaebfff700 1 -- 172.21.15.153:0/1972726557 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbaf000b860 con 0x7fbaf4103e40 2024-03-11T22:54:02.142 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.187+0000 7fbaebfff700 1 -- 172.21.15.153:0/1972726557 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbaf000be90 con 0x7fbaf4103e40 2024-03-11T22:54:02.142 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.187+0000 7fbaebfff700 1 -- 172.21.15.153:0/1972726557 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbaf00103c0 con 0x7fbaf4103e40 2024-03-11T22:54:02.142 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.188+0000 7fbafb50b700 1 -- 172.21.15.153:0/1972726557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbaf4103e40 msgr2=0x7fbaf4104250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:02.142 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.188+0000 7fbafb50b700 1 --2- 172.21.15.153:0/1972726557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbaf4103e40 0x7fbaf4104250 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbaf00098e0 tx=0x7fbaf0009250 comp rx=0 tx=0).stop 2024-03-11T22:54:02.143 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.188+0000 7fbafb50b700 1 -- 172.21.15.153:0/1972726557 shutdown_connections 2024-03-11T22:54:02.143 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.188+0000 7fbafb50b700 1 --2- 172.21.15.153:0/1972726557 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbaf4103e40 0x7fbaf4104250 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:02.143 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.188+0000 7fbafb50b700 1 -- 172.21.15.153:0/1972726557 >> 172.21.15.153:0/1972726557 conn(0x7fbaf40fc580 msgr2=0x7fbaf40fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:02.143 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.188+0000 7fbafb50b700 1 -- 172.21.15.153:0/1972726557 shutdown_connections 2024-03-11T22:54:02.143 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.188+0000 7fbafb50b700 1 -- 172.21.15.153:0/1972726557 wait complete. 2024-03-11T22:54:02.143 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.189+0000 7fbafb50b700 1 Processor -- start 2024-03-11T22:54:02.143 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.189+0000 7fbafb50b700 1 -- start start 2024-03-11T22:54:02.143 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.189+0000 7fbafb50b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbaf4103e40 0x7fbaf4111820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:02.143 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.189+0000 7fbafb50b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf4111d60 con 0x7fbaf4103e40 2024-03-11T22:54:02.143 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.189+0000 7fbaf92a7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbaf4103e40 0x7fbaf4111820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:02.143 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.189+0000 7fbaf92a7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbaf4103e40 0x7fbaf4111820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41146/0 (socket says 172.21.15.153:41146) 2024-03-11T22:54:02.143 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.189+0000 7fbaf92a7700 1 -- 172.21.15.153:0/795865494 learned_addr learned my addr 172.21.15.153:0/795865494 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:54:02.143 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.189+0000 7fbaf92a7700 1 -- 172.21.15.153:0/795865494 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf00094f0 con 0x7fbaf4103e40 2024-03-11T22:54:02.143 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.189+0000 7fbaf92a7700 1 --2- 172.21.15.153:0/795865494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbaf4103e40 0x7fbaf4111820 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0006b00 tx=0x7fbaf0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:02.143 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.189+0000 7fbae9ffb700 1 -- 172.21.15.153:0/795865494 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbaf00109a0 con 0x7fbaf4103e40 2024-03-11T22:54:02.144 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.189+0000 7fbafb50b700 1 -- 172.21.15.153:0/795865494 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaf4111f60 con 0x7fbaf4103e40 2024-03-11T22:54:02.144 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.189+0000 7fbae9ffb700 1 -- 172.21.15.153:0/795865494 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbaf001a430 con 0x7fbaf4103e40 2024-03-11T22:54:02.144 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.189+0000 7fbae9ffb700 1 -- 172.21.15.153:0/795865494 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbaf00245b0 con 0x7fbaf4103e40 2024-03-11T22:54:02.144 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.190+0000 7fbafb50b700 1 -- 172.21.15.153:0/795865494 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaf4112c00 con 0x7fbaf4103e40 2024-03-11T22:54:02.144 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.191+0000 7fbae9ffb700 1 -- 172.21.15.153:0/795865494 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fbaf0024830 con 0x7fbaf4103e40 2024-03-11T22:54:02.144 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.191+0000 7fbae9ffb700 1 --2- 172.21.15.153:0/795865494 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fbae0036790 0x7fbae0038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:02.144 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.191+0000 7fbae9ffb700 1 -- 172.21.15.153:0/795865494 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fbaf001aaa0 con 0x7fbaf4103e40 2024-03-11T22:54:02.144 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.191+0000 7fbaf8aa6700 1 --2- 172.21.15.153:0/795865494 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fbae0036790 0x7fbae0038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:02.144 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.191+0000 7fbafb50b700 1 -- 172.21.15.153:0/795865494 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaf41123f0 con 0x7fbaf4103e40 2024-03-11T22:54:02.144 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.191+0000 7fbaf8aa6700 1 --2- 172.21.15.153:0/795865494 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fbae0036790 0x7fbae0038c40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fbae4009990 tx=0x7fbae4006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:02.144 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.199+0000 7fbae9ffb700 1 -- 172.21.15.153:0/795865494 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7fbaf0018db0 con 0x7fbaf4103e40 2024-03-11T22:54:02.144 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.433+0000 7fbafb50b700 1 -- 172.21.15.153:0/795865494 --> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fbaf4101350 con 0x7fbae0036790 2024-03-11T22:54:02.144 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.438+0000 7fbae9ffb700 1 -- 172.21.15.153:0/795865494 <== mgr.14118 v2:172.21.15.153:6800/1518383615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fbaf4101350 con 0x7fbae0036790 2024-03-11T22:54:02.144 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.440+0000 7fbafb50b700 1 -- 172.21.15.153:0/795865494 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fbae0036790 msgr2=0x7fbae0038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:02.145 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.440+0000 7fbafb50b700 1 --2- 172.21.15.153:0/795865494 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fbae0036790 0x7fbae0038c40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fbae4009990 tx=0x7fbae4006e40 comp rx=0 tx=0).stop 2024-03-11T22:54:02.145 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.440+0000 7fbafb50b700 1 -- 172.21.15.153:0/795865494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbaf4103e40 msgr2=0x7fbaf4111820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:02.145 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.440+0000 7fbafb50b700 1 --2- 172.21.15.153:0/795865494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbaf4103e40 0x7fbaf4111820 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbaf0006b00 tx=0x7fbaf0009250 comp rx=0 tx=0).stop 2024-03-11T22:54:02.145 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.440+0000 7fbafb50b700 1 -- 172.21.15.153:0/795865494 shutdown_connections 2024-03-11T22:54:02.145 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.440+0000 7fbafb50b700 1 --2- 172.21.15.153:0/795865494 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fbae0036790 0x7fbae0038c40 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:02.145 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.440+0000 7fbafb50b700 1 --2- 172.21.15.153:0/795865494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbaf4103e40 0x7fbaf4111820 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:02.145 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.440+0000 7fbafb50b700 1 -- 172.21.15.153:0/795865494 >> 172.21.15.153:0/795865494 conn(0x7fbaf40fc580 msgr2=0x7fbaf40fd390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:02.145 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.440+0000 7fbafb50b700 1 -- 172.21.15.153:0/795865494 shutdown_connections 2024-03-11T22:54:02.145 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:01.440+0000 7fbafb50b700 1 -- 172.21.15.153:0/795865494 wait complete. 2024-03-11T22:54:02.145 INFO:teuthology.orchestra.run.smithi153.stdout:Deploying grafana service with default placement... 2024-03-11T22:54:02.718 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:02 smithi153 ceph-mon[28962]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:54:02.718 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:02 smithi153 ceph-mon[28962]: Saving service prometheus spec with placement count:1 2024-03-11T22:54:02.718 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:02 smithi153 ceph-mon[28962]: from='mgr.14118 172.21.15.153:0/870689843' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:03.742 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-03-11T22:54:03.742 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.935+0000 7fb563cbc700 1 Processor -- start 2024-03-11T22:54:03.742 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.936+0000 7fb563cbc700 1 -- start start 2024-03-11T22:54:03.743 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.936+0000 7fb563cbc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb55c101670 0x7fb55c101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:03.743 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.936+0000 7fb563cbc700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55c101fc0 con 0x7fb55c101670 2024-03-11T22:54:03.743 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.936+0000 7fb561a58700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb55c101670 0x7fb55c101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:03.743 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.936+0000 7fb561a58700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb55c101670 0x7fb55c101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40264/0 (socket says 172.21.15.153:40264) 2024-03-11T22:54:03.743 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.936+0000 7fb561a58700 1 -- 172.21.15.153:0/1240596765 learned_addr learned my addr 172.21.15.153:0/1240596765 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:54:03.743 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.936+0000 7fb561a58700 1 -- 172.21.15.153:0/1240596765 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb55c102100 con 0x7fb55c101670 2024-03-11T22:54:03.743 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.936+0000 7fb561a58700 1 --2- 172.21.15.153:0/1240596765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb55c101670 0x7fb55c101a80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb5580098e0 tx=0x7fb558009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c13aa8a6df5f30d server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:03.743 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.936+0000 7fb560a56700 1 -- 172.21.15.153:0/1240596765 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb55800b860 con 0x7fb55c101670 2024-03-11T22:54:03.743 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.937+0000 7fb560a56700 1 -- 172.21.15.153:0/1240596765 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb55800be90 con 0x7fb55c101670 2024-03-11T22:54:03.743 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.937+0000 7fb560a56700 1 -- 172.21.15.153:0/1240596765 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb5580103c0 con 0x7fb55c101670 2024-03-11T22:54:03.743 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.937+0000 7fb563cbc700 1 -- 172.21.15.153:0/1240596765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb55c101670 msgr2=0x7fb55c101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:03.743 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.937+0000 7fb563cbc700 1 --2- 172.21.15.153:0/1240596765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb55c101670 0x7fb55c101a80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb5580098e0 tx=0x7fb558009250 comp rx=0 tx=0).stop 2024-03-11T22:54:03.743 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.937+0000 7fb563cbc700 1 -- 172.21.15.153:0/1240596765 shutdown_connections 2024-03-11T22:54:03.743 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.937+0000 7fb563cbc700 1 --2- 172.21.15.153:0/1240596765 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb55c101670 0x7fb55c101a80 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:03.743 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.937+0000 7fb563cbc700 1 -- 172.21.15.153:0/1240596765 >> 172.21.15.153:0/1240596765 conn(0x7fb55c0fcdd0 msgr2=0x7fb55c0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:03.744 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.937+0000 7fb563cbc700 1 -- 172.21.15.153:0/1240596765 shutdown_connections 2024-03-11T22:54:03.744 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.937+0000 7fb563cbc700 1 -- 172.21.15.153:0/1240596765 wait complete. 2024-03-11T22:54:03.744 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.938+0000 7fb563cbc700 1 Processor -- start 2024-03-11T22:54:03.744 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.938+0000 7fb563cbc700 1 -- start start 2024-03-11T22:54:03.744 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.938+0000 7fb563cbc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb55c101670 0x7fb55c110760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:03.744 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.938+0000 7fb563cbc700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55c110ca0 con 0x7fb55c101670 2024-03-11T22:54:03.744 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.938+0000 7fb561a58700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb55c101670 0x7fb55c110760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:03.744 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.938+0000 7fb561a58700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb55c101670 0x7fb55c110760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40280/0 (socket says 172.21.15.153:40280) 2024-03-11T22:54:03.744 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.938+0000 7fb561a58700 1 -- 172.21.15.153:0/4088490478 learned_addr learned my addr 172.21.15.153:0/4088490478 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:54:03.744 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.939+0000 7fb561a58700 1 -- 172.21.15.153:0/4088490478 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5580094f0 con 0x7fb55c101670 2024-03-11T22:54:03.744 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.939+0000 7fb561a58700 1 --2- 172.21.15.153:0/4088490478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb55c101670 0x7fb55c110760 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb558006b00 tx=0x7fb558009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:03.744 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.939+0000 7fb5527fc700 1 -- 172.21.15.153:0/4088490478 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb55801b7e0 con 0x7fb55c101670 2024-03-11T22:54:03.744 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.939+0000 7fb5527fc700 1 -- 172.21.15.153:0/4088490478 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb55801be10 con 0x7fb55c101670 2024-03-11T22:54:03.744 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.939+0000 7fb563cbc700 1 -- 172.21.15.153:0/4088490478 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb55c110ea0 con 0x7fb55c101670 2024-03-11T22:54:03.744 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.939+0000 7fb563cbc700 1 -- 172.21.15.153:0/4088490478 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb55c111b40 con 0x7fb55c101670 2024-03-11T22:54:03.745 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.940+0000 7fb5527fc700 1 -- 172.21.15.153:0/4088490478 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb558026620 con 0x7fb55c101670 2024-03-11T22:54:03.745 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.940+0000 7fb5527fc700 1 -- 172.21.15.153:0/4088490478 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fb558026890 con 0x7fb55c101670 2024-03-11T22:54:03.745 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.940+0000 7fb5527fc700 1 --2- 172.21.15.153:0/4088490478 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fb5480367e0 0x7fb548038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:03.745 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.940+0000 7fb563cbc700 1 -- 172.21.15.153:0/4088490478 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5400052e0 con 0x7fb55c101670 2024-03-11T22:54:03.745 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.940+0000 7fb5527fc700 1 -- 172.21.15.153:0/4088490478 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb558010630 con 0x7fb55c101670 2024-03-11T22:54:03.745 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.940+0000 7fb561257700 1 --2- 172.21.15.153:0/4088490478 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fb5480367e0 0x7fb548038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:03.745 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.940+0000 7fb561257700 1 --2- 172.21.15.153:0/4088490478 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fb5480367e0 0x7fb548038c90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb54c009990 tx=0x7fb54c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:03.745 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:02.945+0000 7fb5527fc700 1 -- 172.21.15.153:0/4088490478 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7fb558021070 con 0x7fb55c101670 2024-03-11T22:54:03.745 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:03.150+0000 7fb563cbc700 1 -- 172.21.15.153:0/4088490478 --> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7fb540000bf0 con 0x7fb5480367e0 2024-03-11T22:54:03.745 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:03.155+0000 7fb5527fc700 1 -- 172.21.15.153:0/4088490478 <== mgr.14118 v2:172.21.15.153:6800/1518383615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fb540000bf0 con 0x7fb5480367e0 2024-03-11T22:54:03.745 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:03.157+0000 7fb563cbc700 1 -- 172.21.15.153:0/4088490478 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fb5480367e0 msgr2=0x7fb548038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:03.745 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:03.157+0000 7fb563cbc700 1 --2- 172.21.15.153:0/4088490478 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fb5480367e0 0x7fb548038c90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb54c009990 tx=0x7fb54c006e40 comp rx=0 tx=0).stop 2024-03-11T22:54:03.745 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:03.157+0000 7fb563cbc700 1 -- 172.21.15.153:0/4088490478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb55c101670 msgr2=0x7fb55c110760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:03.745 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:03.157+0000 7fb563cbc700 1 --2- 172.21.15.153:0/4088490478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb55c101670 0x7fb55c110760 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fb558006b00 tx=0x7fb558009250 comp rx=0 tx=0).stop 2024-03-11T22:54:03.745 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:03.157+0000 7fb563cbc700 1 -- 172.21.15.153:0/4088490478 shutdown_connections 2024-03-11T22:54:03.746 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:03.157+0000 7fb563cbc700 1 --2- 172.21.15.153:0/4088490478 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fb5480367e0 0x7fb548038c90 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:03.746 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:03.157+0000 7fb563cbc700 1 --2- 172.21.15.153:0/4088490478 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb55c101670 0x7fb55c110760 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:03.746 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:03.157+0000 7fb563cbc700 1 -- 172.21.15.153:0/4088490478 >> 172.21.15.153:0/4088490478 conn(0x7fb55c0fcdd0 msgr2=0x7fb55c0fdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:03.746 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:03.157+0000 7fb563cbc700 1 -- 172.21.15.153:0/4088490478 shutdown_connections 2024-03-11T22:54:03.746 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:03.157+0000 7fb563cbc700 1 -- 172.21.15.153:0/4088490478 wait complete. 2024-03-11T22:54:03.746 INFO:teuthology.orchestra.run.smithi153.stdout:Deploying node-exporter service with default placement... 2024-03-11T22:54:03.860 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:03 smithi153 ceph-mon[28962]: from='mgr.14118 172.21.15.153:0/870689843' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:03.860 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:03 smithi153 ceph-mon[28962]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:54:03.860 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:03 smithi153 ceph-mon[28962]: Saving service grafana spec with placement count:1 2024-03-11T22:54:03.860 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:03 smithi153 ceph-mon[28962]: from='mgr.14118 172.21.15.153:0/870689843' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:03.860 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:03 smithi153 ceph-mon[28962]: from='mgr.14118 172.21.15.153:0/870689843' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:05.398 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-03-11T22:54:05.404 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.123+0000 7fd2859fd700 1 Processor -- start 2024-03-11T22:54:05.404 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.123+0000 7fd2859fd700 1 -- start start 2024-03-11T22:54:05.404 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.123+0000 7fd2859fd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd28006e270 0x7fd28010cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:05.404 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.123+0000 7fd2859fd700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd28006e680 con 0x7fd28006e270 2024-03-11T22:54:05.404 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.123+0000 7fd27effd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd28006e270 0x7fd28010cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:05.404 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.123+0000 7fd27effd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd28006e270 0x7fd28010cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40294/0 (socket says 172.21.15.153:40294) 2024-03-11T22:54:05.404 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.123+0000 7fd27effd700 1 -- 172.21.15.153:0/3376721222 learned_addr learned my addr 172.21.15.153:0/3376721222 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:54:05.404 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.124+0000 7fd27effd700 1 -- 172.21.15.153:0/3376721222 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd28010d280 con 0x7fd28006e270 2024-03-11T22:54:05.404 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.124+0000 7fd27effd700 1 --2- 172.21.15.153:0/3376721222 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd28006e270 0x7fd28010cd40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd2700098e0 tx=0x7fd270009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=466ee8449ceb8025 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:05.404 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.124+0000 7fd27dffb700 1 -- 172.21.15.153:0/3376721222 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd27000b860 con 0x7fd28006e270 2024-03-11T22:54:05.404 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.124+0000 7fd27dffb700 1 -- 172.21.15.153:0/3376721222 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd27000be90 con 0x7fd28006e270 2024-03-11T22:54:05.404 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.124+0000 7fd27dffb700 1 -- 172.21.15.153:0/3376721222 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2700103c0 con 0x7fd28006e270 2024-03-11T22:54:05.405 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.125+0000 7fd2859fd700 1 -- 172.21.15.153:0/3376721222 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd28006e270 msgr2=0x7fd28010cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:05.405 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.125+0000 7fd2859fd700 1 --2- 172.21.15.153:0/3376721222 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd28006e270 0x7fd28010cd40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd2700098e0 tx=0x7fd270009250 comp rx=0 tx=0).stop 2024-03-11T22:54:05.405 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.125+0000 7fd2859fd700 1 -- 172.21.15.153:0/3376721222 shutdown_connections 2024-03-11T22:54:05.405 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.125+0000 7fd2859fd700 1 --2- 172.21.15.153:0/3376721222 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd28006e270 0x7fd28010cd40 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:05.405 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.125+0000 7fd2859fd700 1 -- 172.21.15.153:0/3376721222 >> 172.21.15.153:0/3376721222 conn(0x7fd280069d90 msgr2=0x7fd28006c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:05.405 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.125+0000 7fd2859fd700 1 -- 172.21.15.153:0/3376721222 shutdown_connections 2024-03-11T22:54:05.405 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.126+0000 7fd2859fd700 1 -- 172.21.15.153:0/3376721222 wait complete. 2024-03-11T22:54:05.405 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.126+0000 7fd2859fd700 1 Processor -- start 2024-03-11T22:54:05.405 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.127+0000 7fd2859fd700 1 -- start start 2024-03-11T22:54:05.405 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.127+0000 7fd2859fd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd28006f500 0x7fd28011cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:05.405 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.127+0000 7fd2859fd700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd28011d4a0 con 0x7fd28006f500 2024-03-11T22:54:05.406 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.127+0000 7fd27effd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd28006f500 0x7fd28011cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:05.406 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.127+0000 7fd27effd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd28006f500 0x7fd28011cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40304/0 (socket says 172.21.15.153:40304) 2024-03-11T22:54:05.406 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.127+0000 7fd27effd700 1 -- 172.21.15.153:0/832719464 learned_addr learned my addr 172.21.15.153:0/832719464 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:54:05.406 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.127+0000 7fd27effd700 1 -- 172.21.15.153:0/832719464 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2700094f0 con 0x7fd28006f500 2024-03-11T22:54:05.406 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.127+0000 7fd27effd700 1 --2- 172.21.15.153:0/832719464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd28006f500 0x7fd28011cf60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd27000b440 tx=0x7fd270009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:05.406 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.128+0000 7fd2677fe700 1 -- 172.21.15.153:0/832719464 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd270011a00 con 0x7fd28006f500 2024-03-11T22:54:05.406 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.128+0000 7fd2677fe700 1 -- 172.21.15.153:0/832719464 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd27001f070 con 0x7fd28006f500 2024-03-11T22:54:05.406 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.128+0000 7fd2677fe700 1 -- 172.21.15.153:0/832719464 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2700269d0 con 0x7fd28006f500 2024-03-11T22:54:05.406 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.128+0000 7fd2859fd700 1 -- 172.21.15.153:0/832719464 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd28011d5e0 con 0x7fd28006f500 2024-03-11T22:54:05.406 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.128+0000 7fd2859fd700 1 -- 172.21.15.153:0/832719464 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd28011da10 con 0x7fd28006f500 2024-03-11T22:54:05.406 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.129+0000 7fd2677fe700 1 -- 172.21.15.153:0/832719464 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fd270026b30 con 0x7fd28006f500 2024-03-11T22:54:05.406 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.129+0000 7fd2677fe700 1 --2- 172.21.15.153:0/832719464 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fd268036790 0x7fd268038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:05.406 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.129+0000 7fd2677fe700 1 -- 172.21.15.153:0/832719464 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fd270021070 con 0x7fd28006f500 2024-03-11T22:54:05.407 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.129+0000 7fd2859fd700 1 -- 172.21.15.153:0/832719464 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd280112dd0 con 0x7fd28006f500 2024-03-11T22:54:05.407 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.129+0000 7fd27e7fc700 1 --2- 172.21.15.153:0/832719464 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fd268036790 0x7fd268038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:05.407 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.129+0000 7fd27e7fc700 1 --2- 172.21.15.153:0/832719464 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fd268036790 0x7fd268038c40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd274009990 tx=0x7fd274006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:05.407 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.134+0000 7fd2677fe700 1 -- 172.21.15.153:0/832719464 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7fd27001b440 con 0x7fd28006f500 2024-03-11T22:54:05.407 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.332+0000 7fd2859fd700 1 -- 172.21.15.153:0/832719464 --> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fd28010dfb0 con 0x7fd268036790 2024-03-11T22:54:05.407 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.336+0000 7fd2677fe700 1 -- 172.21.15.153:0/832719464 <== mgr.14118 v2:172.21.15.153:6800/1518383615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fd28010dfb0 con 0x7fd268036790 2024-03-11T22:54:05.407 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.338+0000 7fd2859fd700 1 -- 172.21.15.153:0/832719464 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fd268036790 msgr2=0x7fd268038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:05.407 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.338+0000 7fd2859fd700 1 --2- 172.21.15.153:0/832719464 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fd268036790 0x7fd268038c40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd274009990 tx=0x7fd274006e40 comp rx=0 tx=0).stop 2024-03-11T22:54:05.407 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.338+0000 7fd2859fd700 1 -- 172.21.15.153:0/832719464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd28006f500 msgr2=0x7fd28011cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:05.407 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.338+0000 7fd2859fd700 1 --2- 172.21.15.153:0/832719464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd28006f500 0x7fd28011cf60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd27000b440 tx=0x7fd270009250 comp rx=0 tx=0).stop 2024-03-11T22:54:05.407 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.338+0000 7fd2859fd700 1 -- 172.21.15.153:0/832719464 shutdown_connections 2024-03-11T22:54:05.407 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.338+0000 7fd2859fd700 1 --2- 172.21.15.153:0/832719464 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fd268036790 0x7fd268038c40 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:05.407 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.338+0000 7fd2859fd700 1 --2- 172.21.15.153:0/832719464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd28006f500 0x7fd28011cf60 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:05.407 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.338+0000 7fd2859fd700 1 -- 172.21.15.153:0/832719464 >> 172.21.15.153:0/832719464 conn(0x7fd280069d90 msgr2=0x7fd28006b550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:05.408 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.338+0000 7fd2859fd700 1 -- 172.21.15.153:0/832719464 shutdown_connections 2024-03-11T22:54:05.408 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:04.338+0000 7fd2859fd700 1 -- 172.21.15.153:0/832719464 wait complete. 2024-03-11T22:54:05.408 INFO:teuthology.orchestra.run.smithi153.stdout:Deploying alertmanager service with default placement... 2024-03-11T22:54:05.513 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:05 smithi153 ceph-mon[28962]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:54:05.513 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:05 smithi153 ceph-mon[28962]: Saving service node-exporter spec with placement * 2024-03-11T22:54:05.513 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:05 smithi153 ceph-mon[28962]: from='mgr.14118 172.21.15.153:0/870689843' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:06.720 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-03-11T22:54:06.720 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.800+0000 7f03b0593700 1 Processor -- start 2024-03-11T22:54:06.720 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.800+0000 7f03b0593700 1 -- start start 2024-03-11T22:54:06.720 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.800+0000 7f03b0593700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03a8078ce0 0x7f03a80771f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:06.720 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.800+0000 7f03b0593700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a80790f0 con 0x7f03a8078ce0 2024-03-11T22:54:06.721 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.800+0000 7f03ae32f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03a8078ce0 0x7f03a80771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:06.721 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.800+0000 7f03ae32f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03a8078ce0 0x7f03a80771f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40312/0 (socket says 172.21.15.153:40312) 2024-03-11T22:54:06.721 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.800+0000 7f03ae32f700 1 -- 172.21.15.153:0/429253993 learned_addr learned my addr 172.21.15.153:0/429253993 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:54:06.721 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.800+0000 7f03ae32f700 1 -- 172.21.15.153:0/429253993 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a8077730 con 0x7f03a8078ce0 2024-03-11T22:54:06.721 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.801+0000 7f03ae32f700 1 --2- 172.21.15.153:0/429253993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03a8078ce0 0x7f03a80771f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f03a40098e0 tx=0x7f03a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd5b4b61f3fac658 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:06.721 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.801+0000 7f03ad32d700 1 -- 172.21.15.153:0/429253993 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03a400b860 con 0x7f03a8078ce0 2024-03-11T22:54:06.721 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.801+0000 7f03ad32d700 1 -- 172.21.15.153:0/429253993 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f03a400be90 con 0x7f03a8078ce0 2024-03-11T22:54:06.721 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.801+0000 7f03ad32d700 1 -- 172.21.15.153:0/429253993 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03a40103c0 con 0x7f03a8078ce0 2024-03-11T22:54:06.721 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.801+0000 7f03b0593700 1 -- 172.21.15.153:0/429253993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03a8078ce0 msgr2=0x7f03a80771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:06.721 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.801+0000 7f03b0593700 1 --2- 172.21.15.153:0/429253993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03a8078ce0 0x7f03a80771f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f03a40098e0 tx=0x7f03a4009250 comp rx=0 tx=0).stop 2024-03-11T22:54:06.721 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.802+0000 7f03b0593700 1 -- 172.21.15.153:0/429253993 shutdown_connections 2024-03-11T22:54:06.721 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.802+0000 7f03b0593700 1 --2- 172.21.15.153:0/429253993 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03a8078ce0 0x7f03a80771f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:06.721 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.802+0000 7f03b0593700 1 -- 172.21.15.153:0/429253993 >> 172.21.15.153:0/429253993 conn(0x7f03a80fded0 msgr2=0x7f03a8100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:06.721 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.802+0000 7f03b0593700 1 -- 172.21.15.153:0/429253993 shutdown_connections 2024-03-11T22:54:06.722 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.802+0000 7f03b0593700 1 -- 172.21.15.153:0/429253993 wait complete. 2024-03-11T22:54:06.722 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.802+0000 7f03b0593700 1 Processor -- start 2024-03-11T22:54:06.722 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.802+0000 7f03b0593700 1 -- start start 2024-03-11T22:54:06.722 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.802+0000 7f03b0593700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03a8078ce0 0x7f03a8113750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:06.722 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.802+0000 7f03b0593700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a8113c90 con 0x7f03a8078ce0 2024-03-11T22:54:06.722 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.802+0000 7f03ae32f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03a8078ce0 0x7f03a8113750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:06.722 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.802+0000 7f03ae32f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03a8078ce0 0x7f03a8113750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40326/0 (socket says 172.21.15.153:40326) 2024-03-11T22:54:06.722 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.802+0000 7f03ae32f700 1 -- 172.21.15.153:0/1849843174 learned_addr learned my addr 172.21.15.153:0/1849843174 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:54:06.722 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.803+0000 7f03ae32f700 1 -- 172.21.15.153:0/1849843174 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a40094f0 con 0x7f03a8078ce0 2024-03-11T22:54:06.722 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.803+0000 7f03ae32f700 1 --2- 172.21.15.153:0/1849843174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03a8078ce0 0x7f03a8113750 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f03a4003080 tx=0x7f03a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:06.722 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.803+0000 7f039effd700 1 -- 172.21.15.153:0/1849843174 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03a4011920 con 0x7f03a8078ce0 2024-03-11T22:54:06.722 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.803+0000 7f039effd700 1 -- 172.21.15.153:0/1849843174 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f03a4010d60 con 0x7f03a8078ce0 2024-03-11T22:54:06.722 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.803+0000 7f03b0593700 1 -- 172.21.15.153:0/1849843174 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03a8113e90 con 0x7f03a8078ce0 2024-03-11T22:54:06.722 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.803+0000 7f039effd700 1 -- 172.21.15.153:0/1849843174 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03a40243f0 con 0x7f03a8078ce0 2024-03-11T22:54:06.722 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.803+0000 7f03b0593700 1 -- 172.21.15.153:0/1849843174 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03a8114b30 con 0x7f03a8078ce0 2024-03-11T22:54:06.723 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.804+0000 7f039effd700 1 -- 172.21.15.153:0/1849843174 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f03a402d3f0 con 0x7f03a8078ce0 2024-03-11T22:54:06.723 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.804+0000 7f03b0593700 1 -- 172.21.15.153:0/1849843174 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03a805f5c0 con 0x7f03a8078ce0 2024-03-11T22:54:06.723 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.804+0000 7f039effd700 1 --2- 172.21.15.153:0/1849843174 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f03940367e0 0x7f0394038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:06.723 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.804+0000 7f039effd700 1 -- 172.21.15.153:0/1849843174 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f03a404a550 con 0x7f03a8078ce0 2024-03-11T22:54:06.723 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.804+0000 7f03adb2e700 1 --2- 172.21.15.153:0/1849843174 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f03940367e0 0x7f0394038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:06.723 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.805+0000 7f03adb2e700 1 --2- 172.21.15.153:0/1849843174 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f03940367e0 0x7f0394038c90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0398009990 tx=0x7f0398006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:06.723 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:05.809+0000 7f039effd700 1 -- 172.21.15.153:0/1849843174 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f03a402b8e0 con 0x7f03a8078ce0 2024-03-11T22:54:06.723 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:06.009+0000 7f03b0593700 1 -- 172.21.15.153:0/1849843174 --> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f03a81001b0 con 0x7f03940367e0 2024-03-11T22:54:06.723 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:06.014+0000 7f039effd700 1 -- 172.21.15.153:0/1849843174 <== mgr.14118 v2:172.21.15.153:6800/1518383615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f03a81001b0 con 0x7f03940367e0 2024-03-11T22:54:06.723 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:06.016+0000 7f03b0593700 1 -- 172.21.15.153:0/1849843174 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f03940367e0 msgr2=0x7f0394038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:06.723 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:06.016+0000 7f03b0593700 1 --2- 172.21.15.153:0/1849843174 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f03940367e0 0x7f0394038c90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f0398009990 tx=0x7f0398006e40 comp rx=0 tx=0).stop 2024-03-11T22:54:06.723 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:06.016+0000 7f03b0593700 1 -- 172.21.15.153:0/1849843174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03a8078ce0 msgr2=0x7f03a8113750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:06.723 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:06.016+0000 7f03b0593700 1 --2- 172.21.15.153:0/1849843174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03a8078ce0 0x7f03a8113750 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f03a4003080 tx=0x7f03a4009250 comp rx=0 tx=0).stop 2024-03-11T22:54:06.723 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:06.016+0000 7f03b0593700 1 -- 172.21.15.153:0/1849843174 shutdown_connections 2024-03-11T22:54:06.723 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:06.016+0000 7f03b0593700 1 --2- 172.21.15.153:0/1849843174 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f03940367e0 0x7f0394038c90 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:06.723 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:06.016+0000 7f03b0593700 1 --2- 172.21.15.153:0/1849843174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03a8078ce0 0x7f03a8113750 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:06.724 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:06.016+0000 7f03b0593700 1 -- 172.21.15.153:0/1849843174 >> 172.21.15.153:0/1849843174 conn(0x7f03a80fded0 msgr2=0x7f03a80ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:06.724 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:06.016+0000 7f03b0593700 1 -- 172.21.15.153:0/1849843174 shutdown_connections 2024-03-11T22:54:06.724 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:06.016+0000 7f03b0593700 1 -- 172.21.15.153:0/1849843174 wait complete. 2024-03-11T22:54:07.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:07 smithi153 ceph-mon[28962]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:54:07.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:07 smithi153 ceph-mon[28962]: Saving service alertmanager spec with placement count:1 2024-03-11T22:54:07.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:07 smithi153 ceph-mon[28962]: from='mgr.14118 172.21.15.153:0/870689843' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:07.893 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.161+0000 7fd4d823f700 1 Processor -- start 2024-03-11T22:54:07.893 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.161+0000 7fd4d823f700 1 -- start start 2024-03-11T22:54:07.893 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.161+0000 7fd4d823f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4d0071480 0x7fd4d0107780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:07.893 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.161+0000 7fd4d823f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d0103170 con 0x7fd4d0071480 2024-03-11T22:54:07.894 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.162+0000 7fd4d5fdb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4d0071480 0x7fd4d0107780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:07.894 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.162+0000 7fd4d5fdb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4d0071480 0x7fd4d0107780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40334/0 (socket says 172.21.15.153:40334) 2024-03-11T22:54:07.894 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.162+0000 7fd4d5fdb700 1 -- 172.21.15.153:0/2188993543 learned_addr learned my addr 172.21.15.153:0/2188993543 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:54:07.894 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.162+0000 7fd4d5fdb700 1 -- 172.21.15.153:0/2188993543 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d0107cc0 con 0x7fd4d0071480 2024-03-11T22:54:07.894 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.162+0000 7fd4d5fdb700 1 --2- 172.21.15.153:0/2188993543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4d0071480 0x7fd4d0107780 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc0098e0 tx=0x7fd4cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11890c3732de0089 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:07.894 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.162+0000 7fd4d4fd9700 1 -- 172.21.15.153:0/2188993543 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4cc00b860 con 0x7fd4d0071480 2024-03-11T22:54:07.894 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.162+0000 7fd4d4fd9700 1 -- 172.21.15.153:0/2188993543 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd4cc00be90 con 0x7fd4d0071480 2024-03-11T22:54:07.894 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.162+0000 7fd4d4fd9700 1 -- 172.21.15.153:0/2188993543 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4cc0103c0 con 0x7fd4d0071480 2024-03-11T22:54:07.894 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.163+0000 7fd4d823f700 1 -- 172.21.15.153:0/2188993543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4d0071480 msgr2=0x7fd4d0107780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:07.894 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.163+0000 7fd4d823f700 1 --2- 172.21.15.153:0/2188993543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4d0071480 0x7fd4d0107780 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc0098e0 tx=0x7fd4cc009250 comp rx=0 tx=0).stop 2024-03-11T22:54:07.894 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.163+0000 7fd4d823f700 1 -- 172.21.15.153:0/2188993543 shutdown_connections 2024-03-11T22:54:07.894 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.163+0000 7fd4d823f700 1 --2- 172.21.15.153:0/2188993543 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4d0071480 0x7fd4d0107780 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:07.894 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.163+0000 7fd4d823f700 1 -- 172.21.15.153:0/2188993543 >> 172.21.15.153:0/2188993543 conn(0x7fd4d00fcdd0 msgr2=0x7fd4d00ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:07.894 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.163+0000 7fd4d823f700 1 -- 172.21.15.153:0/2188993543 shutdown_connections 2024-03-11T22:54:07.895 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.163+0000 7fd4d823f700 1 -- 172.21.15.153:0/2188993543 wait complete. 2024-03-11T22:54:07.895 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.163+0000 7fd4d823f700 1 Processor -- start 2024-03-11T22:54:07.895 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.164+0000 7fd4d823f700 1 -- start start 2024-03-11T22:54:07.895 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.164+0000 7fd4d823f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4d0071480 0x7fd4d01137e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:07.895 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.164+0000 7fd4d823f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d0113d20 con 0x7fd4d0071480 2024-03-11T22:54:07.895 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.164+0000 7fd4d5fdb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4d0071480 0x7fd4d01137e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:07.895 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.164+0000 7fd4d5fdb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4d0071480 0x7fd4d01137e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40350/0 (socket says 172.21.15.153:40350) 2024-03-11T22:54:07.895 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.164+0000 7fd4d5fdb700 1 -- 172.21.15.153:0/160087079 learned_addr learned my addr 172.21.15.153:0/160087079 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:54:07.895 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.164+0000 7fd4d5fdb700 1 -- 172.21.15.153:0/160087079 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4cc0094f0 con 0x7fd4d0071480 2024-03-11T22:54:07.895 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.164+0000 7fd4d5fdb700 1 --2- 172.21.15.153:0/160087079 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4d0071480 0x7fd4d01137e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc0094c0 tx=0x7fd4cc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:07.895 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.164+0000 7fd4c67fc700 1 -- 172.21.15.153:0/160087079 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4cc010890 con 0x7fd4d0071480 2024-03-11T22:54:07.895 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.164+0000 7fd4d823f700 1 -- 172.21.15.153:0/160087079 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4d0113f20 con 0x7fd4d0071480 2024-03-11T22:54:07.895 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.164+0000 7fd4c67fc700 1 -- 172.21.15.153:0/160087079 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd4cc010ec0 con 0x7fd4d0071480 2024-03-11T22:54:07.895 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.164+0000 7fd4c67fc700 1 -- 172.21.15.153:0/160087079 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4cc0243f0 con 0x7fd4d0071480 2024-03-11T22:54:07.895 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.164+0000 7fd4d823f700 1 -- 172.21.15.153:0/160087079 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4d0114bc0 con 0x7fd4d0071480 2024-03-11T22:54:07.896 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.165+0000 7fd4c67fc700 1 -- 172.21.15.153:0/160087079 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fd4cc02d3f0 con 0x7fd4d0071480 2024-03-11T22:54:07.896 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.165+0000 7fd4d823f700 1 -- 172.21.15.153:0/160087079 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4d005f5c0 con 0x7fd4d0071480 2024-03-11T22:54:07.896 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.165+0000 7fd4c67fc700 1 --2- 172.21.15.153:0/160087079 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fd4bc036790 0x7fd4bc038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:07.896 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.165+0000 7fd4c67fc700 1 -- 172.21.15.153:0/160087079 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fd4cc04a410 con 0x7fd4d0071480 2024-03-11T22:54:07.896 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.166+0000 7fd4d57da700 1 --2- 172.21.15.153:0/160087079 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fd4bc036790 0x7fd4bc038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:07.896 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.166+0000 7fd4d57da700 1 --2- 172.21.15.153:0/160087079 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fd4bc036790 0x7fd4bc038c40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd4c0009990 tx=0x7fd4c0006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:07.896 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.170+0000 7fd4c67fc700 1 -- 172.21.15.153:0/160087079 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7fd4cc018b00 con 0x7fd4d0071480 2024-03-11T22:54:07.896 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.355+0000 7fd4d823f700 1 -- 172.21.15.153:0/160087079 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fd4d004bed0 con 0x7fd4d0071480 2024-03-11T22:54:07.896 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.360+0000 7fd4c67fc700 1 -- 172.21.15.153:0/160087079 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fd4cc018b00 con 0x7fd4d0071480 2024-03-11T22:54:07.896 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.362+0000 7fd4d823f700 1 -- 172.21.15.153:0/160087079 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fd4bc036790 msgr2=0x7fd4bc038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:07.896 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.362+0000 7fd4d823f700 1 --2- 172.21.15.153:0/160087079 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fd4bc036790 0x7fd4bc038c40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd4c0009990 tx=0x7fd4c0006e40 comp rx=0 tx=0).stop 2024-03-11T22:54:07.896 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.362+0000 7fd4d823f700 1 -- 172.21.15.153:0/160087079 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4d0071480 msgr2=0x7fd4d01137e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:07.896 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.362+0000 7fd4d823f700 1 --2- 172.21.15.153:0/160087079 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4d0071480 0x7fd4d01137e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd4cc0094c0 tx=0x7fd4cc009250 comp rx=0 tx=0).stop 2024-03-11T22:54:07.896 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.362+0000 7fd4d823f700 1 -- 172.21.15.153:0/160087079 shutdown_connections 2024-03-11T22:54:07.896 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.362+0000 7fd4d823f700 1 --2- 172.21.15.153:0/160087079 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fd4bc036790 0x7fd4bc038c40 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:07.897 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.362+0000 7fd4d823f700 1 --2- 172.21.15.153:0/160087079 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4d0071480 0x7fd4d01137e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:07.897 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.362+0000 7fd4d823f700 1 -- 172.21.15.153:0/160087079 >> 172.21.15.153:0/160087079 conn(0x7fd4d00fcdd0 msgr2=0x7fd4d00fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:07.897 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.362+0000 7fd4d823f700 1 -- 172.21.15.153:0/160087079 shutdown_connections 2024-03-11T22:54:07.897 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:07.362+0000 7fd4d823f700 1 -- 172.21.15.153:0/160087079 wait complete. 2024-03-11T22:54:08.214 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:08 smithi153 ceph-mon[28962]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:54:08.214 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:08 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/160087079' entity='client.admin' 2024-03-11T22:54:09.108 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.307+0000 7fe6b6b06700 1 Processor -- start 2024-03-11T22:54:09.108 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.307+0000 7fe6b6b06700 1 -- start start 2024-03-11T22:54:09.108 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.307+0000 7fe6b6b06700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6b0076c00 0x7fe6b01053a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:09.108 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.307+0000 7fe6b6b06700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b01058e0 con 0x7fe6b0076c00 2024-03-11T22:54:09.109 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.307+0000 7fe6b48a2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6b0076c00 0x7fe6b01053a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:09.109 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.307+0000 7fe6b48a2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6b0076c00 0x7fe6b01053a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40358/0 (socket says 172.21.15.153:40358) 2024-03-11T22:54:09.109 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.307+0000 7fe6b48a2700 1 -- 172.21.15.153:0/758714750 learned_addr learned my addr 172.21.15.153:0/758714750 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:54:09.109 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.307+0000 7fe6b48a2700 1 -- 172.21.15.153:0/758714750 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b0105a20 con 0x7fe6b0076c00 2024-03-11T22:54:09.109 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.308+0000 7fe6b48a2700 1 --2- 172.21.15.153:0/758714750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6b0076c00 0x7fe6b01053a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe6a40098e0 tx=0x7fe6a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68b9235a029cf9ae server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:09.109 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.308+0000 7fe6af7fe700 1 -- 172.21.15.153:0/758714750 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6a400b860 con 0x7fe6b0076c00 2024-03-11T22:54:09.109 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.308+0000 7fe6af7fe700 1 -- 172.21.15.153:0/758714750 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe6a400be90 con 0x7fe6b0076c00 2024-03-11T22:54:09.109 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.308+0000 7fe6af7fe700 1 -- 172.21.15.153:0/758714750 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6a40103c0 con 0x7fe6b0076c00 2024-03-11T22:54:09.109 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.308+0000 7fe6b6b06700 1 -- 172.21.15.153:0/758714750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6b0076c00 msgr2=0x7fe6b01053a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:09.109 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.308+0000 7fe6b6b06700 1 --2- 172.21.15.153:0/758714750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6b0076c00 0x7fe6b01053a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe6a40098e0 tx=0x7fe6a4009250 comp rx=0 tx=0).stop 2024-03-11T22:54:09.109 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.308+0000 7fe6b6b06700 1 -- 172.21.15.153:0/758714750 shutdown_connections 2024-03-11T22:54:09.109 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.308+0000 7fe6b6b06700 1 --2- 172.21.15.153:0/758714750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6b0076c00 0x7fe6b01053a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:09.109 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.308+0000 7fe6b6b06700 1 -- 172.21.15.153:0/758714750 >> 172.21.15.153:0/758714750 conn(0x7fe6b00fcdd0 msgr2=0x7fe6b00ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:09.109 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.308+0000 7fe6b6b06700 1 -- 172.21.15.153:0/758714750 shutdown_connections 2024-03-11T22:54:09.109 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.308+0000 7fe6b6b06700 1 -- 172.21.15.153:0/758714750 wait complete. 2024-03-11T22:54:09.110 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.309+0000 7fe6b6b06700 1 Processor -- start 2024-03-11T22:54:09.110 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.309+0000 7fe6b6b06700 1 -- start start 2024-03-11T22:54:09.110 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.309+0000 7fe6b6b06700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6b0076c00 0x7fe6b0102e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:09.110 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.309+0000 7fe6b6b06700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b01012a0 con 0x7fe6b0076c00 2024-03-11T22:54:09.110 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.309+0000 7fe6b48a2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6b0076c00 0x7fe6b0102e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:09.110 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.309+0000 7fe6b48a2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6b0076c00 0x7fe6b0102e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40368/0 (socket says 172.21.15.153:40368) 2024-03-11T22:54:09.110 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.309+0000 7fe6b48a2700 1 -- 172.21.15.153:0/3879683134 learned_addr learned my addr 172.21.15.153:0/3879683134 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:54:09.110 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.310+0000 7fe6b48a2700 1 -- 172.21.15.153:0/3879683134 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a40094f0 con 0x7fe6b0076c00 2024-03-11T22:54:09.110 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.310+0000 7fe6b48a2700 1 --2- 172.21.15.153:0/3879683134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6b0076c00 0x7fe6b0102e20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe6a40090b0 tx=0x7fe6a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:09.110 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.310+0000 7fe6ad7fa700 1 -- 172.21.15.153:0/3879683134 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6a40118f0 con 0x7fe6b0076c00 2024-03-11T22:54:09.110 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.310+0000 7fe6ad7fa700 1 -- 172.21.15.153:0/3879683134 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe6a4010d10 con 0x7fe6b0076c00 2024-03-11T22:54:09.110 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.310+0000 7fe6ad7fa700 1 -- 172.21.15.153:0/3879683134 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6a4024e70 con 0x7fe6b0076c00 2024-03-11T22:54:09.110 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.310+0000 7fe6b6b06700 1 -- 172.21.15.153:0/3879683134 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6b01014a0 con 0x7fe6b0076c00 2024-03-11T22:54:09.110 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.310+0000 7fe6b6b06700 1 -- 172.21.15.153:0/3879683134 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6b0101930 con 0x7fe6b0076c00 2024-03-11T22:54:09.111 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.311+0000 7fe6ad7fa700 1 -- 172.21.15.153:0/3879683134 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fe6a401a550 con 0x7fe6b0076c00 2024-03-11T22:54:09.111 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.311+0000 7fe6b6b06700 1 -- 172.21.15.153:0/3879683134 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6b010b680 con 0x7fe6b0076c00 2024-03-11T22:54:09.111 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.311+0000 7fe6ad7fa700 1 --2- 172.21.15.153:0/3879683134 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fe6a0036740 0x7fe6a0038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:09.111 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.311+0000 7fe6ad7fa700 1 -- 172.21.15.153:0/3879683134 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fe6a404a690 con 0x7fe6b0076c00 2024-03-11T22:54:09.111 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.311+0000 7fe6affff700 1 --2- 172.21.15.153:0/3879683134 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fe6a0036740 0x7fe6a0038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:09.111 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.311+0000 7fe6affff700 1 --2- 172.21.15.153:0/3879683134 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fe6a0036740 0x7fe6a0038bf0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe69c009990 tx=0x7fe69c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:09.111 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.316+0000 7fe6ad7fa700 1 -- 172.21.15.153:0/3879683134 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7fe6a401a800 con 0x7fe6b0076c00 2024-03-11T22:54:09.111 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.497+0000 7fe6b6b06700 1 -- 172.21.15.153:0/3879683134 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fe6b0064ad0 con 0x7fe6b0076c00 2024-03-11T22:54:09.111 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.501+0000 7fe6ad7fa700 1 -- 172.21.15.153:0/3879683134 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fe6a402c350 con 0x7fe6b0076c00 2024-03-11T22:54:09.111 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.503+0000 7fe6b6b06700 1 -- 172.21.15.153:0/3879683134 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fe6a0036740 msgr2=0x7fe6a0038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:09.111 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.503+0000 7fe6b6b06700 1 --2- 172.21.15.153:0/3879683134 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fe6a0036740 0x7fe6a0038bf0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe69c009990 tx=0x7fe69c006e40 comp rx=0 tx=0).stop 2024-03-11T22:54:09.111 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.503+0000 7fe6b6b06700 1 -- 172.21.15.153:0/3879683134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6b0076c00 msgr2=0x7fe6b0102e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:09.111 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.503+0000 7fe6b6b06700 1 --2- 172.21.15.153:0/3879683134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6b0076c00 0x7fe6b0102e20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe6a40090b0 tx=0x7fe6a4009250 comp rx=0 tx=0).stop 2024-03-11T22:54:09.111 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.504+0000 7fe6b6b06700 1 -- 172.21.15.153:0/3879683134 shutdown_connections 2024-03-11T22:54:09.111 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.504+0000 7fe6b6b06700 1 --2- 172.21.15.153:0/3879683134 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fe6a0036740 0x7fe6a0038bf0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:09.111 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.504+0000 7fe6b6b06700 1 --2- 172.21.15.153:0/3879683134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe6b0076c00 0x7fe6b0102e20 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:09.112 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.504+0000 7fe6b6b06700 1 -- 172.21.15.153:0/3879683134 >> 172.21.15.153:0/3879683134 conn(0x7fe6b00fcdd0 msgr2=0x7fe6b00fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:09.112 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.504+0000 7fe6b6b06700 1 -- 172.21.15.153:0/3879683134 shutdown_connections 2024-03-11T22:54:09.112 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:08.504+0000 7fe6b6b06700 1 -- 172.21.15.153:0/3879683134 wait complete. 2024-03-11T22:54:09.112 INFO:teuthology.orchestra.run.smithi153.stdout:Enabling the dashboard module... 2024-03-11T22:54:09.704 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:09 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/3879683134' entity='client.admin' 2024-03-11T22:54:09.769 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:09 smithi153 ceph-mon[28962]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:54:10.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:10 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/366522854' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-11T22:54:11.099 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.454+0000 7f61527c6700 1 Processor -- start 2024-03-11T22:54:11.099 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.454+0000 7f61527c6700 1 -- start start 2024-03-11T22:54:11.099 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.454+0000 7f61527c6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f614c076c20 0x7f614c105380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:11.099 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.454+0000 7f61527c6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614c1058c0 con 0x7f614c076c20 2024-03-11T22:54:11.099 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.454+0000 7f614bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f614c076c20 0x7f614c105380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:11.099 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.454+0000 7f614bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f614c076c20 0x7f614c105380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40384/0 (socket says 172.21.15.153:40384) 2024-03-11T22:54:11.099 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.454+0000 7f614bfff700 1 -- 172.21.15.153:0/293672280 learned_addr learned my addr 172.21.15.153:0/293672280 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:54:11.099 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.455+0000 7f614bfff700 1 -- 172.21.15.153:0/293672280 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f614c105a00 con 0x7f614c076c20 2024-03-11T22:54:11.099 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.455+0000 7f614bfff700 1 --2- 172.21.15.153:0/293672280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f614c076c20 0x7f614c105380 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f613c0098e0 tx=0x7f613c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe5bdc638858b7f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:11.099 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.455+0000 7f614affd700 1 -- 172.21.15.153:0/293672280 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f613c00b860 con 0x7f614c076c20 2024-03-11T22:54:11.100 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.455+0000 7f614affd700 1 -- 172.21.15.153:0/293672280 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f613c00be90 con 0x7f614c076c20 2024-03-11T22:54:11.100 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.455+0000 7f614affd700 1 -- 172.21.15.153:0/293672280 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f613c0103c0 con 0x7f614c076c20 2024-03-11T22:54:11.100 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.456+0000 7f61527c6700 1 -- 172.21.15.153:0/293672280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f614c076c20 msgr2=0x7f614c105380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:11.100 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.456+0000 7f61527c6700 1 --2- 172.21.15.153:0/293672280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f614c076c20 0x7f614c105380 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f613c0098e0 tx=0x7f613c009250 comp rx=0 tx=0).stop 2024-03-11T22:54:11.100 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.456+0000 7f61527c6700 1 -- 172.21.15.153:0/293672280 shutdown_connections 2024-03-11T22:54:11.100 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.456+0000 7f61527c6700 1 --2- 172.21.15.153:0/293672280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f614c076c20 0x7f614c105380 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:11.100 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.456+0000 7f61527c6700 1 -- 172.21.15.153:0/293672280 >> 172.21.15.153:0/293672280 conn(0x7f614c0fcdd0 msgr2=0x7f614c0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:11.100 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.456+0000 7f61527c6700 1 -- 172.21.15.153:0/293672280 shutdown_connections 2024-03-11T22:54:11.100 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.456+0000 7f61527c6700 1 -- 172.21.15.153:0/293672280 wait complete. 2024-03-11T22:54:11.100 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.456+0000 7f61527c6700 1 Processor -- start 2024-03-11T22:54:11.100 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.456+0000 7f61527c6700 1 -- start start 2024-03-11T22:54:11.100 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.457+0000 7f61527c6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f614c076c20 0x7f614c102e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:11.100 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.457+0000 7f61527c6700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614c101280 con 0x7f614c076c20 2024-03-11T22:54:11.100 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.457+0000 7f614bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f614c076c20 0x7f614c102e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:11.101 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.457+0000 7f614bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f614c076c20 0x7f614c102e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40386/0 (socket says 172.21.15.153:40386) 2024-03-11T22:54:11.101 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.457+0000 7f614bfff700 1 -- 172.21.15.153:0/366522854 learned_addr learned my addr 172.21.15.153:0/366522854 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:54:11.101 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.457+0000 7f614bfff700 1 -- 172.21.15.153:0/366522854 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f613c0094f0 con 0x7f614c076c20 2024-03-11T22:54:11.101 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.457+0000 7f614bfff700 1 --2- 172.21.15.153:0/366522854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f614c076c20 0x7f614c102e00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f613c003080 tx=0x7f613c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:11.101 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.457+0000 7f6148ff9700 1 -- 172.21.15.153:0/366522854 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f613c011920 con 0x7f614c076c20 2024-03-11T22:54:11.101 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.457+0000 7f6148ff9700 1 -- 172.21.15.153:0/366522854 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f613c010d60 con 0x7f614c076c20 2024-03-11T22:54:11.101 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.457+0000 7f61527c6700 1 -- 172.21.15.153:0/366522854 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f614c101480 con 0x7f614c076c20 2024-03-11T22:54:11.101 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.457+0000 7f6148ff9700 1 -- 172.21.15.153:0/366522854 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f613c0243f0 con 0x7f614c076c20 2024-03-11T22:54:11.101 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.457+0000 7f61527c6700 1 -- 172.21.15.153:0/366522854 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f614c101910 con 0x7f614c076c20 2024-03-11T22:54:11.101 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.458+0000 7f6148ff9700 1 -- 172.21.15.153:0/366522854 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f613c02d3f0 con 0x7f614c076c20 2024-03-11T22:54:11.101 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.458+0000 7f6148ff9700 1 --2- 172.21.15.153:0/366522854 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f61340367e0 0x7f6134038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:11.101 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.458+0000 7f6148ff9700 1 -- 172.21.15.153:0/366522854 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f613c04a300 con 0x7f614c076c20 2024-03-11T22:54:11.101 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.458+0000 7f614b7fe700 1 --2- 172.21.15.153:0/366522854 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f61340367e0 0x7f6134038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:11.101 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.458+0000 7f61527c6700 1 -- 172.21.15.153:0/366522854 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f614c10b680 con 0x7f614c076c20 2024-03-11T22:54:11.102 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.459+0000 7f614b7fe700 1 --2- 172.21.15.153:0/366522854 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f61340367e0 0x7f6134038c90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6140009990 tx=0x7f6140006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:11.102 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.463+0000 7f6148ff9700 1 -- 172.21.15.153:0/366522854 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f613c02b8e0 con 0x7f614c076c20 2024-03-11T22:54:11.102 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:09.699+0000 7f61527c6700 1 -- 172.21.15.153:0/366522854 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f614c1067e0 con 0x7f614c076c20 2024-03-11T22:54:11.102 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:10.507+0000 7f6148ff9700 1 -- 172.21.15.153:0/366522854 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f613c0193c0 con 0x7f614c076c20 2024-03-11T22:54:11.102 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:10.508+0000 7f6148ff9700 1 -- 172.21.15.153:0/366522854 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f613c01a750 con 0x7f614c076c20 2024-03-11T22:54:11.102 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:10.511+0000 7f61527c6700 1 -- 172.21.15.153:0/366522854 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f61340367e0 msgr2=0x7f6134038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:11.102 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:10.511+0000 7f61527c6700 1 --2- 172.21.15.153:0/366522854 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f61340367e0 0x7f6134038c90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6140009990 tx=0x7f6140006e40 comp rx=0 tx=0).stop 2024-03-11T22:54:11.102 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:10.511+0000 7f61527c6700 1 -- 172.21.15.153:0/366522854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f614c076c20 msgr2=0x7f614c102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:11.102 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:10.511+0000 7f61527c6700 1 --2- 172.21.15.153:0/366522854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f614c076c20 0x7f614c102e00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f613c003080 tx=0x7f613c009250 comp rx=0 tx=0).stop 2024-03-11T22:54:11.102 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:10.511+0000 7f61527c6700 1 -- 172.21.15.153:0/366522854 shutdown_connections 2024-03-11T22:54:11.102 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:10.511+0000 7f61527c6700 1 --2- 172.21.15.153:0/366522854 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7f61340367e0 0x7f6134038c90 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:11.102 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:10.511+0000 7f61527c6700 1 --2- 172.21.15.153:0/366522854 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f614c076c20 0x7f614c102e00 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:11.102 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:10.511+0000 7f61527c6700 1 -- 172.21.15.153:0/366522854 >> 172.21.15.153:0/366522854 conn(0x7f614c0fcdd0 msgr2=0x7f614c0fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:11.102 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:10.511+0000 7f61527c6700 1 -- 172.21.15.153:0/366522854 shutdown_connections 2024-03-11T22:54:11.103 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:10.511+0000 7f61527c6700 1 -- 172.21.15.153:0/366522854 wait complete. 2024-03-11T22:54:11.747 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:11 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/366522854' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-11T22:54:11.747 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:11 smithi153 ceph-mon[28962]: mgrmap e9: smithi153.nyocep(active, since 23s) 2024-03-11T22:54:12.406 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:54:12.407 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-11T22:54:12.407 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T22:54:12.407 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "active_name": "smithi153.nyocep", 2024-03-11T22:54:12.407 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-11T22:54:12.407 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:54:12.407 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.535+0000 7fd6879fe700 1 Processor -- start 2024-03-11T22:54:12.407 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.536+0000 7fd6879fe700 1 -- start start 2024-03-11T22:54:12.407 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.536+0000 7fd6879fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd680071480 0x7fd680105310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:12.407 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.536+0000 7fd6879fe700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd680105850 con 0x7fd680071480 2024-03-11T22:54:12.407 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.536+0000 7fd68579a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd680071480 0x7fd680105310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:12.407 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.536+0000 7fd68579a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd680071480 0x7fd680105310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:37118/0 (socket says 172.21.15.153:37118) 2024-03-11T22:54:12.407 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.536+0000 7fd68579a700 1 -- 172.21.15.153:0/2621278975 learned_addr learned my addr 172.21.15.153:0/2621278975 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:54:12.408 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.536+0000 7fd68579a700 1 -- 172.21.15.153:0/2621278975 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd680105990 con 0x7fd680071480 2024-03-11T22:54:12.408 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.536+0000 7fd68579a700 1 --2- 172.21.15.153:0/2621278975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd680071480 0x7fd680105310 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd67c0098e0 tx=0x7fd67c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c5cecfdc024e75b server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:12.408 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.537+0000 7fd677fff700 1 -- 172.21.15.153:0/2621278975 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd67c00b860 con 0x7fd680071480 2024-03-11T22:54:12.408 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.537+0000 7fd677fff700 1 -- 172.21.15.153:0/2621278975 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd67c00be90 con 0x7fd680071480 2024-03-11T22:54:12.408 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.537+0000 7fd677fff700 1 -- 172.21.15.153:0/2621278975 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd67c0103c0 con 0x7fd680071480 2024-03-11T22:54:12.408 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.537+0000 7fd6879fe700 1 -- 172.21.15.153:0/2621278975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd680071480 msgr2=0x7fd680105310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:12.408 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.537+0000 7fd6879fe700 1 --2- 172.21.15.153:0/2621278975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd680071480 0x7fd680105310 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd67c0098e0 tx=0x7fd67c009250 comp rx=0 tx=0).stop 2024-03-11T22:54:12.408 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.537+0000 7fd6879fe700 1 -- 172.21.15.153:0/2621278975 shutdown_connections 2024-03-11T22:54:12.408 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.537+0000 7fd6879fe700 1 --2- 172.21.15.153:0/2621278975 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd680071480 0x7fd680105310 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:12.408 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.537+0000 7fd6879fe700 1 -- 172.21.15.153:0/2621278975 >> 172.21.15.153:0/2621278975 conn(0x7fd6800fcd80 msgr2=0x7fd6800ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:12.408 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.537+0000 7fd6879fe700 1 -- 172.21.15.153:0/2621278975 shutdown_connections 2024-03-11T22:54:12.408 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.537+0000 7fd6879fe700 1 -- 172.21.15.153:0/2621278975 wait complete. 2024-03-11T22:54:12.408 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.538+0000 7fd6879fe700 1 Processor -- start 2024-03-11T22:54:12.408 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.538+0000 7fd6879fe700 1 -- start start 2024-03-11T22:54:12.408 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.538+0000 7fd6879fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd680071480 0x7fd680111540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:12.409 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.538+0000 7fd6879fe700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd680111a80 con 0x7fd680071480 2024-03-11T22:54:12.409 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.538+0000 7fd68579a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd680071480 0x7fd680111540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:12.409 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.539+0000 7fd68579a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd680071480 0x7fd680111540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:37130/0 (socket says 172.21.15.153:37130) 2024-03-11T22:54:12.409 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.539+0000 7fd68579a700 1 -- 172.21.15.153:0/4105748708 learned_addr learned my addr 172.21.15.153:0/4105748708 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:54:12.409 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.539+0000 7fd68579a700 1 -- 172.21.15.153:0/4105748708 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd67c0094f0 con 0x7fd680071480 2024-03-11T22:54:12.409 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.539+0000 7fd68579a700 1 --2- 172.21.15.153:0/4105748708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd680071480 0x7fd680111540 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd67c003080 tx=0x7fd67c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:12.409 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.539+0000 7fd675ffb700 1 -- 172.21.15.153:0/4105748708 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd67c011920 con 0x7fd680071480 2024-03-11T22:54:12.409 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.539+0000 7fd675ffb700 1 -- 172.21.15.153:0/4105748708 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd67c010d60 con 0x7fd680071480 2024-03-11T22:54:12.409 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.539+0000 7fd675ffb700 1 -- 172.21.15.153:0/4105748708 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd67c0243f0 con 0x7fd680071480 2024-03-11T22:54:12.409 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.539+0000 7fd6879fe700 1 -- 172.21.15.153:0/4105748708 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd680111c80 con 0x7fd680071480 2024-03-11T22:54:12.409 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.539+0000 7fd6879fe700 1 -- 172.21.15.153:0/4105748708 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd680112920 con 0x7fd680071480 2024-03-11T22:54:12.409 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.540+0000 7fd675ffb700 1 -- 172.21.15.153:0/4105748708 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fd67c01a430 con 0x7fd680071480 2024-03-11T22:54:12.409 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.540+0000 7fd6879fe700 1 -- 172.21.15.153:0/4105748708 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd68005f5c0 con 0x7fd680071480 2024-03-11T22:54:12.410 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.541+0000 7fd675ffb700 1 --2- 172.21.15.153:0/4105748708 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fd66c0367e0 0x7fd66c038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:12.410 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.541+0000 7fd675ffb700 1 -- 172.21.15.153:0/4105748708 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fd67c04a360 con 0x7fd680071480 2024-03-11T22:54:12.410 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.541+0000 7fd684f99700 1 -- 172.21.15.153:0/4105748708 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fd66c0367e0 msgr2=0x7fd66c038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/1518383615 2024-03-11T22:54:12.410 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.541+0000 7fd684f99700 1 --2- 172.21.15.153:0/4105748708 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fd66c0367e0 0x7fd66c038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T22:54:12.410 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.545+0000 7fd675ffb700 1 -- 172.21.15.153:0/4105748708 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7fd67c02b8e0 con 0x7fd680071480 2024-03-11T22:54:12.410 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.742+0000 7fd684f99700 1 -- 172.21.15.153:0/4105748708 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fd66c0367e0 msgr2=0x7fd66c038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/1518383615 2024-03-11T22:54:12.410 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.742+0000 7fd684f99700 1 --2- 172.21.15.153:0/4105748708 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fd66c0367e0 0x7fd66c038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-11T22:54:12.410 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.787+0000 7fd6879fe700 1 -- 172.21.15.153:0/4105748708 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fd680112c60 con 0x7fd680071480 2024-03-11T22:54:12.410 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.787+0000 7fd675ffb700 1 -- 172.21.15.153:0/4105748708 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7fd67c02b2f0 con 0x7fd680071480 2024-03-11T22:54:12.410 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.789+0000 7fd6879fe700 1 -- 172.21.15.153:0/4105748708 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fd66c0367e0 msgr2=0x7fd66c038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T22:54:12.410 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.789+0000 7fd6879fe700 1 --2- 172.21.15.153:0/4105748708 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fd66c0367e0 0x7fd66c038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:12.410 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.789+0000 7fd6879fe700 1 -- 172.21.15.153:0/4105748708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd680071480 msgr2=0x7fd680111540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:12.410 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.789+0000 7fd6879fe700 1 --2- 172.21.15.153:0/4105748708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd680071480 0x7fd680111540 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd67c003080 tx=0x7fd67c009250 comp rx=0 tx=0).stop 2024-03-11T22:54:12.411 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.789+0000 7fd6879fe700 1 -- 172.21.15.153:0/4105748708 shutdown_connections 2024-03-11T22:54:12.411 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.789+0000 7fd6879fe700 1 --2- 172.21.15.153:0/4105748708 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fd66c0367e0 0x7fd66c038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:12.411 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.789+0000 7fd6879fe700 1 --2- 172.21.15.153:0/4105748708 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd680071480 0x7fd680111540 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:12.411 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.789+0000 7fd6879fe700 1 -- 172.21.15.153:0/4105748708 >> 172.21.15.153:0/4105748708 conn(0x7fd6800fcd80 msgr2=0x7fd6800fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:12.411 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.790+0000 7fd6879fe700 1 -- 172.21.15.153:0/4105748708 shutdown_connections 2024-03-11T22:54:12.411 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:11.790+0000 7fd6879fe700 1 -- 172.21.15.153:0/4105748708 wait complete. 2024-03-11T22:54:12.411 INFO:teuthology.orchestra.run.smithi153.stdout:Waiting for the mgr to restart... 2024-03-11T22:54:12.411 INFO:teuthology.orchestra.run.smithi153.stdout:Waiting for mgr epoch 9... 2024-03-11T22:54:12.640 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:12 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/4105748708' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-11T22:54:17.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:17 smithi153 ceph-mon[28962]: Active manager daemon smithi153.nyocep restarted 2024-03-11T22:54:17.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:17 smithi153 ceph-mon[28962]: Activating manager daemon smithi153.nyocep 2024-03-11T22:54:17.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:17 smithi153 ceph-mon[28962]: osdmap e3: 0 total, 0 up, 0 in 2024-03-11T22:54:17.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:17 smithi153 ceph-mon[28962]: mgrmap e10: smithi153.nyocep(active, starting, since 0.00343284s) 2024-03-11T22:54:17.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:17 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-03-11T22:54:17.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:17 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mgr metadata", "who": "smithi153.nyocep", "id": "smithi153.nyocep"}]: dispatch 2024-03-11T22:54:17.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:17 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:54:17.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:17 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:54:17.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:17 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:54:17.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:17 smithi153 ceph-mon[28962]: Manager daemon smithi153.nyocep is now available 2024-03-11T22:54:17.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:17 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:54:17.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:17 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:54:18.450 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:18 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.nyocep/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:54:18.450 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:18 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.nyocep/trash_purge_schedule"}]: dispatch 2024-03-11T22:54:18.450 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:18 smithi153 ceph-mon[28962]: [11/Mar/2024:22:54:17] ENGINE Bus STARTING 2024-03-11T22:54:18.450 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:18 smithi153 ceph-mon[28962]: [11/Mar/2024:22:54:17] ENGINE Serving on https://172.21.15.153:7150 2024-03-11T22:54:18.450 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:18 smithi153 ceph-mon[28962]: [11/Mar/2024:22:54:17] ENGINE Bus STARTED 2024-03-11T22:54:18.450 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:18 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:18.450 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:18 smithi153 ceph-mon[28962]: mgrmap e11: smithi153.nyocep(active, since 1.00589s) 2024-03-11T22:54:18.735 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:54:18.735 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-11T22:54:18.735 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-11T22:54:18.735 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:54:18.735 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.829+0000 7fb80ea35700 1 Processor -- start 2024-03-11T22:54:18.735 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.829+0000 7fb80ea35700 1 -- start start 2024-03-11T22:54:18.735 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.829+0000 7fb80ea35700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb808078d20 0x7fb808077230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:18.735 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.829+0000 7fb80ea35700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb808079130 con 0x7fb808078d20 2024-03-11T22:54:18.735 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.829+0000 7fb807fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb808078d20 0x7fb808077230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:18.736 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.829+0000 7fb807fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb808078d20 0x7fb808077230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:37132/0 (socket says 172.21.15.153:37132) 2024-03-11T22:54:18.736 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.829+0000 7fb807fff700 1 -- 172.21.15.153:0/2433483541 learned_addr learned my addr 172.21.15.153:0/2433483541 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:54:18.736 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.829+0000 7fb807fff700 1 -- 172.21.15.153:0/2433483541 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb808077770 con 0x7fb808078d20 2024-03-11T22:54:18.736 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.830+0000 7fb807fff700 1 --2- 172.21.15.153:0/2433483541 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb808078d20 0x7fb808077230 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb7f80098e0 tx=0x7fb7f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e39586794d126b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:18.736 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.830+0000 7fb806ffd700 1 -- 172.21.15.153:0/2433483541 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7f800b860 con 0x7fb808078d20 2024-03-11T22:54:18.736 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.830+0000 7fb806ffd700 1 -- 172.21.15.153:0/2433483541 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f800be90 con 0x7fb808078d20 2024-03-11T22:54:18.736 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.830+0000 7fb806ffd700 1 -- 172.21.15.153:0/2433483541 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7f80103c0 con 0x7fb808078d20 2024-03-11T22:54:18.736 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.830+0000 7fb80ea35700 1 -- 172.21.15.153:0/2433483541 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb808078d20 msgr2=0x7fb808077230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:18.736 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.830+0000 7fb80ea35700 1 --2- 172.21.15.153:0/2433483541 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb808078d20 0x7fb808077230 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb7f80098e0 tx=0x7fb7f8009250 comp rx=0 tx=0).stop 2024-03-11T22:54:18.736 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.831+0000 7fb80ea35700 1 -- 172.21.15.153:0/2433483541 shutdown_connections 2024-03-11T22:54:18.736 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.831+0000 7fb80ea35700 1 --2- 172.21.15.153:0/2433483541 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb808078d20 0x7fb808077230 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:18.736 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.831+0000 7fb80ea35700 1 -- 172.21.15.153:0/2433483541 >> 172.21.15.153:0/2433483541 conn(0x7fb8080fded0 msgr2=0x7fb808100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:18.736 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.831+0000 7fb80ea35700 1 -- 172.21.15.153:0/2433483541 shutdown_connections 2024-03-11T22:54:18.736 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.831+0000 7fb80ea35700 1 -- 172.21.15.153:0/2433483541 wait complete. 2024-03-11T22:54:18.736 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.831+0000 7fb80ea35700 1 Processor -- start 2024-03-11T22:54:18.737 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.831+0000 7fb80ea35700 1 -- start start 2024-03-11T22:54:18.737 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.832+0000 7fb80ea35700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb808078d20 0x7fb808113790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:18.737 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.832+0000 7fb80ea35700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb808113cd0 con 0x7fb808078d20 2024-03-11T22:54:18.737 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.832+0000 7fb807fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb808078d20 0x7fb808113790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:18.737 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.832+0000 7fb807fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb808078d20 0x7fb808113790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:37140/0 (socket says 172.21.15.153:37140) 2024-03-11T22:54:18.737 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.832+0000 7fb807fff700 1 -- 172.21.15.153:0/1545190772 learned_addr learned my addr 172.21.15.153:0/1545190772 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:54:18.737 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.832+0000 7fb807fff700 1 -- 172.21.15.153:0/1545190772 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f80094f0 con 0x7fb808078d20 2024-03-11T22:54:18.737 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.832+0000 7fb807fff700 1 --2- 172.21.15.153:0/1545190772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb808078d20 0x7fb808113790 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8003080 tx=0x7fb7f8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:18.737 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.832+0000 7fb804ff9700 1 -- 172.21.15.153:0/1545190772 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7f8011920 con 0x7fb808078d20 2024-03-11T22:54:18.737 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.832+0000 7fb804ff9700 1 -- 172.21.15.153:0/1545190772 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb7f8010d60 con 0x7fb808078d20 2024-03-11T22:54:18.737 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.832+0000 7fb80ea35700 1 -- 172.21.15.153:0/1545190772 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb808113ed0 con 0x7fb808078d20 2024-03-11T22:54:18.737 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.832+0000 7fb804ff9700 1 -- 172.21.15.153:0/1545190772 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7f80243f0 con 0x7fb808078d20 2024-03-11T22:54:18.737 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.832+0000 7fb80ea35700 1 -- 172.21.15.153:0/1545190772 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb808114b70 con 0x7fb808078d20 2024-03-11T22:54:18.737 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.833+0000 7fb804ff9700 1 -- 172.21.15.153:0/1545190772 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fb7f802d3f0 con 0x7fb808078d20 2024-03-11T22:54:18.738 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.833+0000 7fb804ff9700 1 --2- 172.21.15.153:0/1545190772 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fb7f00367e0 0x7fb7f0038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:18.738 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.833+0000 7fb80ea35700 1 -- 172.21.15.153:0/1545190772 --> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb80805f5c0 con 0x7fb7f00367e0 2024-03-11T22:54:18.738 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.833+0000 7fb804ff9700 1 -- 172.21.15.153:0/1545190772 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb7f804b3d0 con 0x7fb808078d20 2024-03-11T22:54:18.738 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.833+0000 7fb8077fe700 1 -- 172.21.15.153:0/1545190772 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fb7f00367e0 msgr2=0x7fb7f0038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/1518383615 2024-03-11T22:54:18.738 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:12.833+0000 7fb8077fe700 1 --2- 172.21.15.153:0/1545190772 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fb7f00367e0 0x7fb7f0038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T22:54:18.738 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:13.034+0000 7fb8077fe700 1 -- 172.21.15.153:0/1545190772 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fb7f00367e0 msgr2=0x7fb7f0038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/1518383615 2024-03-11T22:54:18.738 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:13.034+0000 7fb8077fe700 1 --2- 172.21.15.153:0/1545190772 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fb7f00367e0 0x7fb7f0038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-11T22:54:18.738 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:13.435+0000 7fb8077fe700 1 -- 172.21.15.153:0/1545190772 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fb7f00367e0 msgr2=0x7fb7f0038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/1518383615 2024-03-11T22:54:18.738 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:13.435+0000 7fb8077fe700 1 --2- 172.21.15.153:0/1545190772 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fb7f00367e0 0x7fb7f0038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-11T22:54:18.738 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:14.236+0000 7fb8077fe700 1 -- 172.21.15.153:0/1545190772 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fb7f00367e0 msgr2=0x7fb7f0038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/1518383615 2024-03-11T22:54:18.738 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:14.236+0000 7fb8077fe700 1 --2- 172.21.15.153:0/1545190772 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fb7f00367e0 0x7fb7f0038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-11T22:54:18.738 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:15.837+0000 7fb8077fe700 1 -- 172.21.15.153:0/1545190772 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fb7f00367e0 msgr2=0x7fb7f0038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/1518383615 2024-03-11T22:54:18.739 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:15.837+0000 7fb8077fe700 1 --2- 172.21.15.153:0/1545190772 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fb7f00367e0 0x7fb7f0038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-11T22:54:18.739 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:17.067+0000 7fb804ff9700 1 -- 172.21.15.153:0/1545190772 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mgrmap(e 10) v1 ==== 43148+0+0 (secure 0 0 0) 0x7fb7f801a430 con 0x7fb808078d20 2024-03-11T22:54:18.739 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:17.067+0000 7fb804ff9700 1 -- 172.21.15.153:0/1545190772 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fb7f00367e0 msgr2=0x7fb7f0038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T22:54:18.739 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:17.067+0000 7fb804ff9700 1 --2- 172.21.15.153:0/1545190772 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fb7f00367e0 0x7fb7f0038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:18.739 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:18.068+0000 7fb804ff9700 1 -- 172.21.15.153:0/1545190772 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mgrmap(e 11) v1 ==== 43275+0+0 (secure 0 0 0) 0x7fb7f80193c0 con 0x7fb808078d20 2024-03-11T22:54:18.739 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:18.068+0000 7fb804ff9700 1 --2- 172.21.15.153:0/1545190772 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7fb7f0039ff0 0x7fb7f003c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:18.739 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:18.069+0000 7fb804ff9700 1 -- 172.21.15.153:0/1545190772 --> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb80805f5c0 con 0x7fb7f0039ff0 2024-03-11T22:54:18.739 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:18.069+0000 7fb8077fe700 1 --2- 172.21.15.153:0/1545190772 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7fb7f0039ff0 0x7fb7f003c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:18.739 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:18.070+0000 7fb8077fe700 1 --2- 172.21.15.153:0/1545190772 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7fb7f0039ff0 0x7fb7f003c3d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc0039d0 tx=0x7fb7fc0092f0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:18.739 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:18.070+0000 7fb804ff9700 1 -- 172.21.15.153:0/1545190772 <== mgr.14160 v2:172.21.15.153:6800/4212724556 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7fb80805f5c0 con 0x7fb7f0039ff0 2024-03-11T22:54:18.739 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:18.076+0000 7fb80ea35700 1 -- 172.21.15.153:0/1545190772 --> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fb808114360 con 0x7fb7f0039ff0 2024-03-11T22:54:18.739 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:18.076+0000 7fb804ff9700 1 -- 172.21.15.153:0/1545190772 <== mgr.14160 v2:172.21.15.153:6800/4212724556 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fb808114360 con 0x7fb7f0039ff0 2024-03-11T22:54:18.739 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:18.077+0000 7fb80ea35700 1 -- 172.21.15.153:0/1545190772 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7fb7f0039ff0 msgr2=0x7fb7f003c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:18.739 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:18.077+0000 7fb80ea35700 1 --2- 172.21.15.153:0/1545190772 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7fb7f0039ff0 0x7fb7f003c3d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc0039d0 tx=0x7fb7fc0092f0 comp rx=0 tx=0).stop 2024-03-11T22:54:18.739 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:18.077+0000 7fb80ea35700 1 -- 172.21.15.153:0/1545190772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb808078d20 msgr2=0x7fb808113790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:18.740 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:18.077+0000 7fb80ea35700 1 --2- 172.21.15.153:0/1545190772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb808078d20 0x7fb808113790 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb7f8003080 tx=0x7fb7f8009250 comp rx=0 tx=0).stop 2024-03-11T22:54:18.740 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:18.077+0000 7fb80ea35700 1 -- 172.21.15.153:0/1545190772 shutdown_connections 2024-03-11T22:54:18.740 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:18.077+0000 7fb80ea35700 1 --2- 172.21.15.153:0/1545190772 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7fb7f0039ff0 0x7fb7f003c3d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:18.740 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:18.077+0000 7fb80ea35700 1 --2- 172.21.15.153:0/1545190772 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb808078d20 0x7fb808113790 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:18.740 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:18.077+0000 7fb80ea35700 1 --2- 172.21.15.153:0/1545190772 >> [v2:172.21.15.153:6800/1518383615,v1:172.21.15.153:6801/1518383615] conn(0x7fb7f00367e0 0x7fb7f0038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:18.740 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:18.077+0000 7fb80ea35700 1 -- 172.21.15.153:0/1545190772 >> 172.21.15.153:0/1545190772 conn(0x7fb8080fded0 msgr2=0x7fb8080ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:18.740 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:18.077+0000 7fb80ea35700 1 -- 172.21.15.153:0/1545190772 shutdown_connections 2024-03-11T22:54:18.740 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:18.077+0000 7fb80ea35700 1 -- 172.21.15.153:0/1545190772 wait complete. 2024-03-11T22:54:18.740 INFO:teuthology.orchestra.run.smithi153.stdout:mgr epoch 9 is available 2024-03-11T22:54:18.740 INFO:teuthology.orchestra.run.smithi153.stdout:Generating a dashboard self-signed certificate... 2024-03-11T22:54:19.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:19 smithi153 ceph-mon[28962]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-11T22:54:19.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:19 smithi153 ceph-mon[28962]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-11T22:54:19.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:19 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:20.090 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-11T22:54:20.090 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.178+0000 7f031c6f8700 1 Processor -- start 2024-03-11T22:54:20.090 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.178+0000 7f031c6f8700 1 -- start start 2024-03-11T22:54:20.090 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.178+0000 7f031c6f8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0314101670 0x7f0314101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:20.091 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.178+0000 7f031c6f8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0314101fc0 con 0x7f0314101670 2024-03-11T22:54:20.091 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.178+0000 7f031a494700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0314101670 0x7f0314101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:20.091 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.178+0000 7f031a494700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0314101670 0x7f0314101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:37216/0 (socket says 172.21.15.153:37216) 2024-03-11T22:54:20.091 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.178+0000 7f031a494700 1 -- 172.21.15.153:0/2679693062 learned_addr learned my addr 172.21.15.153:0/2679693062 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:54:20.091 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.179+0000 7f031a494700 1 -- 172.21.15.153:0/2679693062 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0314102100 con 0x7f0314101670 2024-03-11T22:54:20.091 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.179+0000 7f031a494700 1 --2- 172.21.15.153:0/2679693062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0314101670 0x7f0314101a80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f03100098e0 tx=0x7f0310009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6be029d4e435cd78 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:20.091 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.179+0000 7f0319492700 1 -- 172.21.15.153:0/2679693062 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f031000b860 con 0x7f0314101670 2024-03-11T22:54:20.091 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.179+0000 7f0319492700 1 -- 172.21.15.153:0/2679693062 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f031000be90 con 0x7f0314101670 2024-03-11T22:54:20.091 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.179+0000 7f0319492700 1 -- 172.21.15.153:0/2679693062 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03100103c0 con 0x7f0314101670 2024-03-11T22:54:20.091 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.179+0000 7f031c6f8700 1 -- 172.21.15.153:0/2679693062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0314101670 msgr2=0x7f0314101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:20.091 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.179+0000 7f031c6f8700 1 --2- 172.21.15.153:0/2679693062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0314101670 0x7f0314101a80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f03100098e0 tx=0x7f0310009250 comp rx=0 tx=0).stop 2024-03-11T22:54:20.091 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.180+0000 7f031c6f8700 1 -- 172.21.15.153:0/2679693062 shutdown_connections 2024-03-11T22:54:20.091 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.180+0000 7f031c6f8700 1 --2- 172.21.15.153:0/2679693062 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0314101670 0x7f0314101a80 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:20.091 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.180+0000 7f031c6f8700 1 -- 172.21.15.153:0/2679693062 >> 172.21.15.153:0/2679693062 conn(0x7f03140fcdd0 msgr2=0x7f03140ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:20.092 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.180+0000 7f031c6f8700 1 -- 172.21.15.153:0/2679693062 shutdown_connections 2024-03-11T22:54:20.092 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.180+0000 7f031c6f8700 1 -- 172.21.15.153:0/2679693062 wait complete. 2024-03-11T22:54:20.092 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.180+0000 7f031c6f8700 1 Processor -- start 2024-03-11T22:54:20.092 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.180+0000 7f031c6f8700 1 -- start start 2024-03-11T22:54:20.092 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.181+0000 7f031c6f8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0314101670 0x7f0314110440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:20.092 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.181+0000 7f031c6f8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0314110980 con 0x7f0314101670 2024-03-11T22:54:20.092 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.181+0000 7f031a494700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0314101670 0x7f0314110440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:20.092 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.181+0000 7f031a494700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0314101670 0x7f0314110440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:37218/0 (socket says 172.21.15.153:37218) 2024-03-11T22:54:20.092 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.181+0000 7f031a494700 1 -- 172.21.15.153:0/1197637188 learned_addr learned my addr 172.21.15.153:0/1197637188 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:54:20.092 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.181+0000 7f031a494700 1 -- 172.21.15.153:0/1197637188 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03100094f0 con 0x7f0314101670 2024-03-11T22:54:20.092 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.181+0000 7f031a494700 1 --2- 172.21.15.153:0/1197637188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0314101670 0x7f0314110440 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0310006b00 tx=0x7f0310009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:20.092 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.181+0000 7f030affd700 1 -- 172.21.15.153:0/1197637188 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03100109a0 con 0x7f0314101670 2024-03-11T22:54:20.092 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.181+0000 7f030affd700 1 -- 172.21.15.153:0/1197637188 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f031001a430 con 0x7f0314101670 2024-03-11T22:54:20.092 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.181+0000 7f031c6f8700 1 -- 172.21.15.153:0/1197637188 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0314110b80 con 0x7f0314101670 2024-03-11T22:54:20.092 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.181+0000 7f031c6f8700 1 -- 172.21.15.153:0/1197637188 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0314111820 con 0x7f0314101670 2024-03-11T22:54:20.093 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.181+0000 7f030affd700 1 -- 172.21.15.153:0/1197637188 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03100245b0 con 0x7f0314101670 2024-03-11T22:54:20.093 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.182+0000 7f031c6f8700 1 -- 172.21.15.153:0/1197637188 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02f80052e0 con 0x7f0314101670 2024-03-11T22:54:20.093 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.182+0000 7f030affd700 1 -- 172.21.15.153:0/1197637188 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 11) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f03100247e0 con 0x7f0314101670 2024-03-11T22:54:20.093 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.183+0000 7f030affd700 1 --2- 172.21.15.153:0/1197637188 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f0300036710 0x7f0300038bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:20.093 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.183+0000 7f030affd700 1 -- 172.21.15.153:0/1197637188 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f031004a540 con 0x7f0314101670 2024-03-11T22:54:20.093 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.183+0000 7f0319c93700 1 --2- 172.21.15.153:0/1197637188 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f0300036710 0x7f0300038bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:20.093 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.183+0000 7f0319c93700 1 --2- 172.21.15.153:0/1197637188 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f0300036710 0x7f0300038bc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0304009990 tx=0x7f0304006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:20.093 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.187+0000 7f030affd700 1 -- 172.21.15.153:0/1197637188 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f0310018d90 con 0x7f0314101670 2024-03-11T22:54:20.093 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.386+0000 7f031c6f8700 1 -- 172.21.15.153:0/1197637188 --> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f02f8000bf0 con 0x7f0300036710 2024-03-11T22:54:20.093 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.481+0000 7f030affd700 1 -- 172.21.15.153:0/1197637188 <== mgr.14160 v2:172.21.15.153:6800/4212724556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f02f8000bf0 con 0x7f0300036710 2024-03-11T22:54:20.093 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.483+0000 7f031c6f8700 1 -- 172.21.15.153:0/1197637188 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f0300036710 msgr2=0x7f0300038bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:20.093 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.483+0000 7f031c6f8700 1 --2- 172.21.15.153:0/1197637188 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f0300036710 0x7f0300038bc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0304009990 tx=0x7f0304006e40 comp rx=0 tx=0).stop 2024-03-11T22:54:20.093 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.483+0000 7f031c6f8700 1 -- 172.21.15.153:0/1197637188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0314101670 msgr2=0x7f0314110440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:20.093 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.483+0000 7f031c6f8700 1 --2- 172.21.15.153:0/1197637188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0314101670 0x7f0314110440 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0310006b00 tx=0x7f0310009250 comp rx=0 tx=0).stop 2024-03-11T22:54:20.094 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.483+0000 7f031c6f8700 1 -- 172.21.15.153:0/1197637188 shutdown_connections 2024-03-11T22:54:20.094 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.483+0000 7f031c6f8700 1 --2- 172.21.15.153:0/1197637188 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f0300036710 0x7f0300038bc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:20.094 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.483+0000 7f031c6f8700 1 --2- 172.21.15.153:0/1197637188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0314101670 0x7f0314110440 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:20.094 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.483+0000 7f031c6f8700 1 -- 172.21.15.153:0/1197637188 >> 172.21.15.153:0/1197637188 conn(0x7f03140fcdd0 msgr2=0x7f03140fdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:20.094 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.483+0000 7f031c6f8700 1 -- 172.21.15.153:0/1197637188 shutdown_connections 2024-03-11T22:54:20.094 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:19.483+0000 7f031c6f8700 1 -- 172.21.15.153:0/1197637188 wait complete. 2024-03-11T22:54:20.094 INFO:teuthology.orchestra.run.smithi153.stdout:Creating initial admin user... 2024-03-11T22:54:20.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:20 smithi153 ceph-mon[28962]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:54:20.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:20 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:20.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:20 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:20.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:20 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:20.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:20 smithi153 ceph-mon[28962]: mgrmap e12: smithi153.nyocep(active, since 2s) 2024-03-11T22:54:22.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:21 smithi153 ceph-mon[28962]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:54:22.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:21 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:22.227 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$0q/sUT6LtJgOSRRrzMaDuePctQGY.abGjtfAX0sAx3uYItf.9UOpO", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710197660, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-11T22:54:22.227 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.489+0000 7f1559211700 1 Processor -- start 2024-03-11T22:54:22.227 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.489+0000 7f1559211700 1 -- start start 2024-03-11T22:54:22.228 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.489+0000 7f1559211700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1554102c60 0x7f1554103070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:22.228 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.489+0000 7f1559211700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15541035b0 con 0x7f1554102c60 2024-03-11T22:54:22.228 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.489+0000 7f1552d9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1554102c60 0x7f1554103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:22.228 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.490+0000 7f1552d9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1554102c60 0x7f1554103070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:37226/0 (socket says 172.21.15.153:37226) 2024-03-11T22:54:22.228 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.490+0000 7f1552d9d700 1 -- 172.21.15.153:0/1072912167 learned_addr learned my addr 172.21.15.153:0/1072912167 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:54:22.228 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.490+0000 7f1552d9d700 1 -- 172.21.15.153:0/1072912167 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15541036f0 con 0x7f1554102c60 2024-03-11T22:54:22.228 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.490+0000 7f1552d9d700 1 --2- 172.21.15.153:0/1072912167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1554102c60 0x7f1554103070 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f15440098e0 tx=0x7f1544009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73d7bb03e110da3a server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:22.228 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.490+0000 7f1551d9b700 1 -- 172.21.15.153:0/1072912167 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f154400b860 con 0x7f1554102c60 2024-03-11T22:54:22.228 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.490+0000 7f1551d9b700 1 -- 172.21.15.153:0/1072912167 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f154400be90 con 0x7f1554102c60 2024-03-11T22:54:22.228 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.490+0000 7f1551d9b700 1 -- 172.21.15.153:0/1072912167 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15440103c0 con 0x7f1554102c60 2024-03-11T22:54:22.228 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.491+0000 7f1559211700 1 -- 172.21.15.153:0/1072912167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1554102c60 msgr2=0x7f1554103070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:22.228 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.491+0000 7f1559211700 1 --2- 172.21.15.153:0/1072912167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1554102c60 0x7f1554103070 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f15440098e0 tx=0x7f1544009250 comp rx=0 tx=0).stop 2024-03-11T22:54:22.228 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.491+0000 7f1559211700 1 -- 172.21.15.153:0/1072912167 shutdown_connections 2024-03-11T22:54:22.228 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.491+0000 7f1559211700 1 --2- 172.21.15.153:0/1072912167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1554102c60 0x7f1554103070 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:22.228 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.491+0000 7f1559211700 1 -- 172.21.15.153:0/1072912167 >> 172.21.15.153:0/1072912167 conn(0x7f15540fc580 msgr2=0x7f15540fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:22.229 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.491+0000 7f1559211700 1 -- 172.21.15.153:0/1072912167 shutdown_connections 2024-03-11T22:54:22.229 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.491+0000 7f1559211700 1 -- 172.21.15.153:0/1072912167 wait complete. 2024-03-11T22:54:22.229 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.492+0000 7f1559211700 1 Processor -- start 2024-03-11T22:54:22.229 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.492+0000 7f1559211700 1 -- start start 2024-03-11T22:54:22.229 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.492+0000 7f1559211700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1554102c60 0x7f1554107a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:22.229 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.492+0000 7f1559211700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1554105f10 con 0x7f1554102c60 2024-03-11T22:54:22.229 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.492+0000 7f1552d9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1554102c60 0x7f1554107a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:22.229 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.492+0000 7f1552d9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1554102c60 0x7f1554107a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:37230/0 (socket says 172.21.15.153:37230) 2024-03-11T22:54:22.229 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.492+0000 7f1552d9d700 1 -- 172.21.15.153:0/4031687894 learned_addr learned my addr 172.21.15.153:0/4031687894 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:54:22.229 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.492+0000 7f1552d9d700 1 -- 172.21.15.153:0/4031687894 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15440094f0 con 0x7f1554102c60 2024-03-11T22:54:22.229 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.493+0000 7f1552d9d700 1 --2- 172.21.15.153:0/4031687894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1554102c60 0x7f1554107a90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1544006b00 tx=0x7f1544009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:22.229 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.493+0000 7f153b7fe700 1 -- 172.21.15.153:0/4031687894 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15440109a0 con 0x7f1554102c60 2024-03-11T22:54:22.229 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.493+0000 7f1559211700 1 -- 172.21.15.153:0/4031687894 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1554106110 con 0x7f1554102c60 2024-03-11T22:54:22.229 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.493+0000 7f153b7fe700 1 -- 172.21.15.153:0/4031687894 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f154401a430 con 0x7f1554102c60 2024-03-11T22:54:22.229 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.493+0000 7f153b7fe700 1 -- 172.21.15.153:0/4031687894 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15440245b0 con 0x7f1554102c60 2024-03-11T22:54:22.230 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.493+0000 7f1559211700 1 -- 172.21.15.153:0/4031687894 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15541065a0 con 0x7f1554102c60 2024-03-11T22:54:22.230 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.494+0000 7f153b7fe700 1 -- 172.21.15.153:0/4031687894 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f1544024830 con 0x7f1554102c60 2024-03-11T22:54:22.230 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.494+0000 7f1559211700 1 -- 172.21.15.153:0/4031687894 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f155410b770 con 0x7f1554102c60 2024-03-11T22:54:22.230 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.494+0000 7f153b7fe700 1 --2- 172.21.15.153:0/4031687894 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f153c036830 0x7f153c038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:22.230 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.494+0000 7f153b7fe700 1 -- 172.21.15.153:0/4031687894 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f154404a9c0 con 0x7f1554102c60 2024-03-11T22:54:22.230 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.495+0000 7f155259c700 1 --2- 172.21.15.153:0/4031687894 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f153c036830 0x7f153c038ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:22.230 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.495+0000 7f155259c700 1 --2- 172.21.15.153:0/4031687894 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f153c036830 0x7f153c038ce0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1548009990 tx=0x7f1548006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:22.230 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.500+0000 7f153b7fe700 1 -- 172.21.15.153:0/4031687894 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f1544018530 con 0x7f1554102c60 2024-03-11T22:54:22.230 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.709+0000 7f1559211700 1 -- 172.21.15.153:0/4031687894 --> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f155405f3e0 con 0x7f153c036830 2024-03-11T22:54:22.230 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.933+0000 7f153b7fe700 1 -- 172.21.15.153:0/4031687894 <== mgr.14160 v2:172.21.15.153:6800/4212724556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f155405f3e0 con 0x7f153c036830 2024-03-11T22:54:22.230 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.935+0000 7f1559211700 1 -- 172.21.15.153:0/4031687894 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f153c036830 msgr2=0x7f153c038ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:22.230 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.935+0000 7f1559211700 1 --2- 172.21.15.153:0/4031687894 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f153c036830 0x7f153c038ce0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1548009990 tx=0x7f1548006e40 comp rx=0 tx=0).stop 2024-03-11T22:54:22.230 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.936+0000 7f1559211700 1 -- 172.21.15.153:0/4031687894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1554102c60 msgr2=0x7f1554107a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:22.230 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.936+0000 7f1559211700 1 --2- 172.21.15.153:0/4031687894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1554102c60 0x7f1554107a90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1544006b00 tx=0x7f1544009250 comp rx=0 tx=0).stop 2024-03-11T22:54:22.231 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.936+0000 7f1559211700 1 -- 172.21.15.153:0/4031687894 shutdown_connections 2024-03-11T22:54:22.231 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.936+0000 7f1559211700 1 --2- 172.21.15.153:0/4031687894 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f153c036830 0x7f153c038ce0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:22.231 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.936+0000 7f1559211700 1 --2- 172.21.15.153:0/4031687894 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1554102c60 0x7f1554107a90 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:22.231 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.936+0000 7f1559211700 1 -- 172.21.15.153:0/4031687894 >> 172.21.15.153:0/4031687894 conn(0x7f15540fc580 msgr2=0x7f15540fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:22.231 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.936+0000 7f1559211700 1 -- 172.21.15.153:0/4031687894 shutdown_connections 2024-03-11T22:54:22.231 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:20.936+0000 7f1559211700 1 -- 172.21.15.153:0/4031687894 wait complete. 2024-03-11T22:54:22.231 INFO:teuthology.orchestra.run.smithi153.stdout:Fetching dashboard port number... 2024-03-11T22:54:23.184 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:22 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/812801861' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-11T22:54:23.400 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 8443 2024-03-11T22:54:23.400 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.657+0000 7f4d04c31700 1 Processor -- start 2024-03-11T22:54:23.400 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.657+0000 7f4d04c31700 1 -- start start 2024-03-11T22:54:23.400 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.658+0000 7f4d04c31700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d00078ce0 0x7f4d000771f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:23.400 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.658+0000 7f4d04c31700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d000790f0 con 0x7f4d00078ce0 2024-03-11T22:54:23.400 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.658+0000 7f4cfe59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d00078ce0 0x7f4d000771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:23.400 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.658+0000 7f4cfe59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d00078ce0 0x7f4d000771f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52612/0 (socket says 172.21.15.153:52612) 2024-03-11T22:54:23.400 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.658+0000 7f4cfe59c700 1 -- 172.21.15.153:0/363605491 learned_addr learned my addr 172.21.15.153:0/363605491 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:54:23.400 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.658+0000 7f4cfe59c700 1 -- 172.21.15.153:0/363605491 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d00077730 con 0x7f4d00078ce0 2024-03-11T22:54:23.401 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.658+0000 7f4cfe59c700 1 --2- 172.21.15.153:0/363605491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d00078ce0 0x7f4d000771f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4ce80098e0 tx=0x7f4ce8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fdf3d34f2da32aa server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:23.401 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.659+0000 7f4cfd59a700 1 -- 172.21.15.153:0/363605491 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ce800b860 con 0x7f4d00078ce0 2024-03-11T22:54:23.401 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.659+0000 7f4cfd59a700 1 -- 172.21.15.153:0/363605491 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4ce800be90 con 0x7f4d00078ce0 2024-03-11T22:54:23.401 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.659+0000 7f4cfd59a700 1 -- 172.21.15.153:0/363605491 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ce80103c0 con 0x7f4d00078ce0 2024-03-11T22:54:23.401 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.659+0000 7f4d04c31700 1 -- 172.21.15.153:0/363605491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d00078ce0 msgr2=0x7f4d000771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:23.401 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.659+0000 7f4d04c31700 1 --2- 172.21.15.153:0/363605491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d00078ce0 0x7f4d000771f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4ce80098e0 tx=0x7f4ce8009250 comp rx=0 tx=0).stop 2024-03-11T22:54:23.401 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.659+0000 7f4d04c31700 1 -- 172.21.15.153:0/363605491 shutdown_connections 2024-03-11T22:54:23.401 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.659+0000 7f4d04c31700 1 --2- 172.21.15.153:0/363605491 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d00078ce0 0x7f4d000771f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:23.401 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.659+0000 7f4d04c31700 1 -- 172.21.15.153:0/363605491 >> 172.21.15.153:0/363605491 conn(0x7f4d000fded0 msgr2=0x7f4d00100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:23.401 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.659+0000 7f4d04c31700 1 -- 172.21.15.153:0/363605491 shutdown_connections 2024-03-11T22:54:23.401 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.659+0000 7f4d04c31700 1 -- 172.21.15.153:0/363605491 wait complete. 2024-03-11T22:54:23.401 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.660+0000 7f4d04c31700 1 Processor -- start 2024-03-11T22:54:23.401 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.660+0000 7f4d04c31700 1 -- start start 2024-03-11T22:54:23.401 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.660+0000 7f4d04c31700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d00078ce0 0x7f4d00113820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:23.402 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.660+0000 7f4d04c31700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d00113d60 con 0x7f4d00078ce0 2024-03-11T22:54:23.402 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.660+0000 7f4cfe59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d00078ce0 0x7f4d00113820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:23.402 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.660+0000 7f4cfe59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d00078ce0 0x7f4d00113820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52618/0 (socket says 172.21.15.153:52618) 2024-03-11T22:54:23.402 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.660+0000 7f4cfe59c700 1 -- 172.21.15.153:0/812801861 learned_addr learned my addr 172.21.15.153:0/812801861 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:54:23.402 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.660+0000 7f4cfe59c700 1 -- 172.21.15.153:0/812801861 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce80094f0 con 0x7f4d00078ce0 2024-03-11T22:54:23.402 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.660+0000 7f4cfe59c700 1 --2- 172.21.15.153:0/812801861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d00078ce0 0x7f4d00113820 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4ce80094c0 tx=0x7f4ce8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:23.402 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.661+0000 7f4cf6ffd700 1 -- 172.21.15.153:0/812801861 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ce8010890 con 0x7f4d00078ce0 2024-03-11T22:54:23.402 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.661+0000 7f4d04c31700 1 -- 172.21.15.153:0/812801861 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d00113f60 con 0x7f4d00078ce0 2024-03-11T22:54:23.402 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.661+0000 7f4cf6ffd700 1 -- 172.21.15.153:0/812801861 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4ce8010ec0 con 0x7f4d00078ce0 2024-03-11T22:54:23.402 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.661+0000 7f4cf6ffd700 1 -- 172.21.15.153:0/812801861 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ce80243f0 con 0x7f4d00078ce0 2024-03-11T22:54:23.402 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.661+0000 7f4d04c31700 1 -- 172.21.15.153:0/812801861 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d00114c00 con 0x7f4d00078ce0 2024-03-11T22:54:23.402 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.661+0000 7f4cf6ffd700 1 -- 172.21.15.153:0/812801861 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f4ce802d3f0 con 0x7f4d00078ce0 2024-03-11T22:54:23.402 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.661+0000 7f4d04c31700 1 -- 172.21.15.153:0/812801861 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d0005f5c0 con 0x7f4d00078ce0 2024-03-11T22:54:23.402 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.662+0000 7f4cf6ffd700 1 --2- 172.21.15.153:0/812801861 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f4cec036830 0x7f4cec038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:23.402 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.662+0000 7f4cf6ffd700 1 -- 172.21.15.153:0/812801861 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4ce804a4e0 con 0x7f4d00078ce0 2024-03-11T22:54:23.403 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.662+0000 7f4cfdd9b700 1 --2- 172.21.15.153:0/812801861 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f4cec036830 0x7f4cec038ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:23.403 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.662+0000 7f4cfdd9b700 1 --2- 172.21.15.153:0/812801861 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f4cec036830 0x7f4cec038ce0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4cf0009990 tx=0x7f4cf0006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:23.403 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.666+0000 7f4cf6ffd700 1 -- 172.21.15.153:0/812801861 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f4ce8018b00 con 0x7f4d00078ce0 2024-03-11T22:54:23.403 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.848+0000 7f4d04c31700 1 -- 172.21.15.153:0/812801861 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f4d0004bed0 con 0x7f4d00078ce0 2024-03-11T22:54:23.403 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.848+0000 7f4cf6ffd700 1 -- 172.21.15.153:0/812801861 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f4ce8018b00 con 0x7f4d00078ce0 2024-03-11T22:54:23.403 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.850+0000 7f4d04c31700 1 -- 172.21.15.153:0/812801861 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f4cec036830 msgr2=0x7f4cec038ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:23.403 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.850+0000 7f4d04c31700 1 --2- 172.21.15.153:0/812801861 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f4cec036830 0x7f4cec038ce0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4cf0009990 tx=0x7f4cf0006e40 comp rx=0 tx=0).stop 2024-03-11T22:54:23.403 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.850+0000 7f4d04c31700 1 -- 172.21.15.153:0/812801861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d00078ce0 msgr2=0x7f4d00113820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:23.403 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.850+0000 7f4d04c31700 1 --2- 172.21.15.153:0/812801861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d00078ce0 0x7f4d00113820 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4ce80094c0 tx=0x7f4ce8009250 comp rx=0 tx=0).stop 2024-03-11T22:54:23.403 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.850+0000 7f4d04c31700 1 -- 172.21.15.153:0/812801861 shutdown_connections 2024-03-11T22:54:23.403 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.850+0000 7f4d04c31700 1 --2- 172.21.15.153:0/812801861 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f4cec036830 0x7f4cec038ce0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:23.403 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.850+0000 7f4d04c31700 1 --2- 172.21.15.153:0/812801861 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4d00078ce0 0x7f4d00113820 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:23.403 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.850+0000 7f4d04c31700 1 -- 172.21.15.153:0/812801861 >> 172.21.15.153:0/812801861 conn(0x7f4d000fded0 msgr2=0x7f4d000ff610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:23.403 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.851+0000 7f4d04c31700 1 -- 172.21.15.153:0/812801861 shutdown_connections 2024-03-11T22:54:23.403 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:22.851+0000 7f4d04c31700 1 -- 172.21.15.153:0/812801861 wait complete. 2024-03-11T22:54:23.407 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T22:54:23.407 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout disabled 2024-03-11T22:54:23.414 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T22:54:23.414 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout inactive 2024-03-11T22:54:23.414 INFO:teuthology.orchestra.run.smithi153.stdout:firewalld.service is not enabled 2024-03-11T22:54:23.414 INFO:teuthology.orchestra.run.smithi153.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-11T22:54:23.416 INFO:teuthology.orchestra.run.smithi153.stdout:Ceph Dashboard is now available at: 2024-03-11T22:54:23.416 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:54:23.416 INFO:teuthology.orchestra.run.smithi153.stdout: URL: https://smithi153.front.sepia.ceph.com:8443/ 2024-03-11T22:54:23.416 INFO:teuthology.orchestra.run.smithi153.stdout: User: admin 2024-03-11T22:54:23.416 INFO:teuthology.orchestra.run.smithi153.stdout: Password: kgstnykevv 2024-03-11T22:54:23.416 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:54:23.416 INFO:teuthology.orchestra.run.smithi153.stdout:Saving cluster configuration to /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/config directory 2024-03-11T22:54:23.417 INFO:teuthology.orchestra.run.smithi153.stdout:Enabling autotune for osd_memory_target 2024-03-11T22:54:24.598 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.821+0000 7f1d549e1700 1 Processor -- start 2024-03-11T22:54:24.598 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.821+0000 7f1d549e1700 1 -- start start 2024-03-11T22:54:24.598 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.821+0000 7f1d549e1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d50102770 0x7f1d50102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:24.598 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.821+0000 7f1d549e1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d501030c0 con 0x7f1d50102770 2024-03-11T22:54:24.598 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.821+0000 7f1d4e59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d50102770 0x7f1d50102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:24.598 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.821+0000 7f1d4e59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d50102770 0x7f1d50102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52630/0 (socket says 172.21.15.153:52630) 2024-03-11T22:54:24.598 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.821+0000 7f1d4e59c700 1 -- 172.21.15.153:0/4004583577 learned_addr learned my addr 172.21.15.153:0/4004583577 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:54:24.598 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.822+0000 7f1d4e59c700 1 -- 172.21.15.153:0/4004583577 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d50103200 con 0x7f1d50102770 2024-03-11T22:54:24.599 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.822+0000 7f1d4e59c700 1 --2- 172.21.15.153:0/4004583577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d50102770 0x7f1d50102b80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1d400098e0 tx=0x7f1d40009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8867b4ffff9ba976 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:24.599 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.822+0000 7f1d4dd9b700 1 -- 172.21.15.153:0/4004583577 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d4000b860 con 0x7f1d50102770 2024-03-11T22:54:24.599 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.822+0000 7f1d4dd9b700 1 -- 172.21.15.153:0/4004583577 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1d4000be90 con 0x7f1d50102770 2024-03-11T22:54:24.599 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.822+0000 7f1d4dd9b700 1 -- 172.21.15.153:0/4004583577 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d400103c0 con 0x7f1d50102770 2024-03-11T22:54:24.599 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.823+0000 7f1d549e1700 1 -- 172.21.15.153:0/4004583577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d50102770 msgr2=0x7f1d50102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:24.599 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.823+0000 7f1d549e1700 1 --2- 172.21.15.153:0/4004583577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d50102770 0x7f1d50102b80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1d400098e0 tx=0x7f1d40009250 comp rx=0 tx=0).stop 2024-03-11T22:54:24.599 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.823+0000 7f1d549e1700 1 -- 172.21.15.153:0/4004583577 shutdown_connections 2024-03-11T22:54:24.599 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.823+0000 7f1d549e1700 1 --2- 172.21.15.153:0/4004583577 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d50102770 0x7f1d50102b80 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:24.599 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.823+0000 7f1d549e1700 1 -- 172.21.15.153:0/4004583577 >> 172.21.15.153:0/4004583577 conn(0x7f1d500fded0 msgr2=0x7f1d50100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:24.599 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.823+0000 7f1d549e1700 1 -- 172.21.15.153:0/4004583577 shutdown_connections 2024-03-11T22:54:24.599 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.823+0000 7f1d549e1700 1 -- 172.21.15.153:0/4004583577 wait complete. 2024-03-11T22:54:24.599 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.823+0000 7f1d549e1700 1 Processor -- start 2024-03-11T22:54:24.599 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.823+0000 7f1d549e1700 1 -- start start 2024-03-11T22:54:24.599 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.824+0000 7f1d549e1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d50102770 0x7f1d501115c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:24.600 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.824+0000 7f1d549e1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d50111b00 con 0x7f1d50102770 2024-03-11T22:54:24.600 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.824+0000 7f1d4e59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d50102770 0x7f1d501115c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:24.600 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.824+0000 7f1d4e59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d50102770 0x7f1d501115c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52632/0 (socket says 172.21.15.153:52632) 2024-03-11T22:54:24.600 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.824+0000 7f1d4e59c700 1 -- 172.21.15.153:0/1480062238 learned_addr learned my addr 172.21.15.153:0/1480062238 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:54:24.600 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.824+0000 7f1d4e59c700 1 -- 172.21.15.153:0/1480062238 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d400094f0 con 0x7f1d50102770 2024-03-11T22:54:24.600 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.824+0000 7f1d4e59c700 1 --2- 172.21.15.153:0/1480062238 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d50102770 0x7f1d501115c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1d40011680 tx=0x7f1d40009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:24.600 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.824+0000 7f1d477fe700 1 -- 172.21.15.153:0/1480062238 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d40010890 con 0x7f1d50102770 2024-03-11T22:54:24.600 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.824+0000 7f1d549e1700 1 -- 172.21.15.153:0/1480062238 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d50111d00 con 0x7f1d50102770 2024-03-11T22:54:24.600 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.824+0000 7f1d477fe700 1 -- 172.21.15.153:0/1480062238 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1d40010ec0 con 0x7f1d50102770 2024-03-11T22:54:24.600 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.824+0000 7f1d477fe700 1 -- 172.21.15.153:0/1480062238 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1d400243f0 con 0x7f1d50102770 2024-03-11T22:54:24.600 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.824+0000 7f1d549e1700 1 -- 172.21.15.153:0/1480062238 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d501129a0 con 0x7f1d50102770 2024-03-11T22:54:24.600 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.825+0000 7f1d477fe700 1 -- 172.21.15.153:0/1480062238 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f1d4002d3f0 con 0x7f1d50102770 2024-03-11T22:54:24.600 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.825+0000 7f1d477fe700 1 --2- 172.21.15.153:0/1480062238 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f1d3c0367e0 0x7f1d3c038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:24.600 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.825+0000 7f1d477fe700 1 -- 172.21.15.153:0/1480062238 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f1d4004a3b0 con 0x7f1d50102770 2024-03-11T22:54:24.601 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.825+0000 7f1d549e1700 1 -- 172.21.15.153:0/1480062238 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d5005f5c0 con 0x7f1d50102770 2024-03-11T22:54:24.601 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.825+0000 7f1d45d9b700 1 --2- 172.21.15.153:0/1480062238 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f1d3c0367e0 0x7f1d3c038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:24.601 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.826+0000 7f1d45d9b700 1 --2- 172.21.15.153:0/1480062238 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f1d3c0367e0 0x7f1d3c038c90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1d38009990 tx=0x7f1d38006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:24.601 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.830+0000 7f1d477fe700 1 -- 172.21.15.153:0/1480062238 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f1d400193c0 con 0x7f1d50102770 2024-03-11T22:54:24.601 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:23.935+0000 7f1d477fe700 1 -- 172.21.15.153:0/1480062238 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f1d4002dd50 con 0x7f1d50102770 2024-03-11T22:54:24.601 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:24.013+0000 7f1d549e1700 1 -- 172.21.15.153:0/1480062238 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f1d5004bed0 con 0x7f1d50102770 2024-03-11T22:54:24.601 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:24.013+0000 7f1d477fe700 1 -- 172.21.15.153:0/1480062238 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f1d40006d70 con 0x7f1d50102770 2024-03-11T22:54:24.601 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:24.015+0000 7f1d549e1700 1 -- 172.21.15.153:0/1480062238 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f1d3c0367e0 msgr2=0x7f1d3c038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:24.601 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:24.015+0000 7f1d549e1700 1 --2- 172.21.15.153:0/1480062238 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f1d3c0367e0 0x7f1d3c038c90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1d38009990 tx=0x7f1d38006e40 comp rx=0 tx=0).stop 2024-03-11T22:54:24.601 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:24.015+0000 7f1d549e1700 1 -- 172.21.15.153:0/1480062238 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d50102770 msgr2=0x7f1d501115c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:24.601 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:24.015+0000 7f1d549e1700 1 --2- 172.21.15.153:0/1480062238 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d50102770 0x7f1d501115c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1d40011680 tx=0x7f1d40009250 comp rx=0 tx=0).stop 2024-03-11T22:54:24.601 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:24.015+0000 7f1d549e1700 1 -- 172.21.15.153:0/1480062238 shutdown_connections 2024-03-11T22:54:24.601 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:24.015+0000 7f1d549e1700 1 --2- 172.21.15.153:0/1480062238 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f1d3c0367e0 0x7f1d3c038c90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:24.601 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:24.015+0000 7f1d549e1700 1 --2- 172.21.15.153:0/1480062238 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1d50102770 0x7f1d501115c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:24.602 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:24.015+0000 7f1d549e1700 1 -- 172.21.15.153:0/1480062238 >> 172.21.15.153:0/1480062238 conn(0x7f1d500fded0 msgr2=0x7f1d500ff5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:24.602 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:24.015+0000 7f1d549e1700 1 -- 172.21.15.153:0/1480062238 shutdown_connections 2024-03-11T22:54:24.602 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:24.015+0000 7f1d549e1700 1 -- 172.21.15.153:0/1480062238 wait complete. 2024-03-11T22:54:25.018 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:24 smithi153 ceph-mon[28962]: mgrmap e13: smithi153.nyocep(active, since 6s) 2024-03-11T22:54:25.821 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.020+0000 7f4cc4cdf700 1 Processor -- start 2024-03-11T22:54:25.822 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.020+0000 7f4cc4cdf700 1 -- start start 2024-03-11T22:54:25.822 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.021+0000 7f4cc4cdf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cc0102780 0x7f4cc0102b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:25.822 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.021+0000 7f4cc4cdf700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc01030d0 con 0x7f4cc0102780 2024-03-11T22:54:25.822 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.021+0000 7f4cbe59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cc0102780 0x7f4cc0102b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:25.822 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.021+0000 7f4cbe59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cc0102780 0x7f4cc0102b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52638/0 (socket says 172.21.15.153:52638) 2024-03-11T22:54:25.822 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.021+0000 7f4cbe59c700 1 -- 172.21.15.153:0/1810005137 learned_addr learned my addr 172.21.15.153:0/1810005137 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:54:25.822 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.021+0000 7f4cbe59c700 1 -- 172.21.15.153:0/1810005137 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc0103210 con 0x7f4cc0102780 2024-03-11T22:54:25.822 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.021+0000 7f4cbe59c700 1 --2- 172.21.15.153:0/1810005137 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cc0102780 0x7f4cc0102b90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4cb00098e0 tx=0x7f4cb0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bc35828dc96d1c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:25.822 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.021+0000 7f4cbd59a700 1 -- 172.21.15.153:0/1810005137 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4cb000b860 con 0x7f4cc0102780 2024-03-11T22:54:25.822 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.021+0000 7f4cbd59a700 1 -- 172.21.15.153:0/1810005137 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4cb000be90 con 0x7f4cc0102780 2024-03-11T22:54:25.822 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.021+0000 7f4cbd59a700 1 -- 172.21.15.153:0/1810005137 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4cb00103c0 con 0x7f4cc0102780 2024-03-11T22:54:25.823 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.022+0000 7f4cc4cdf700 1 -- 172.21.15.153:0/1810005137 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cc0102780 msgr2=0x7f4cc0102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:25.823 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.022+0000 7f4cc4cdf700 1 --2- 172.21.15.153:0/1810005137 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cc0102780 0x7f4cc0102b90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4cb00098e0 tx=0x7f4cb0009250 comp rx=0 tx=0).stop 2024-03-11T22:54:25.823 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.022+0000 7f4cc4cdf700 1 -- 172.21.15.153:0/1810005137 shutdown_connections 2024-03-11T22:54:25.823 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.022+0000 7f4cc4cdf700 1 --2- 172.21.15.153:0/1810005137 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cc0102780 0x7f4cc0102b90 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:25.823 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.022+0000 7f4cc4cdf700 1 -- 172.21.15.153:0/1810005137 >> 172.21.15.153:0/1810005137 conn(0x7f4cc00fdf00 msgr2=0x7f4cc0100310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:25.823 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.022+0000 7f4cc4cdf700 1 -- 172.21.15.153:0/1810005137 shutdown_connections 2024-03-11T22:54:25.823 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.022+0000 7f4cc4cdf700 1 -- 172.21.15.153:0/1810005137 wait complete. 2024-03-11T22:54:25.823 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.022+0000 7f4cc4cdf700 1 Processor -- start 2024-03-11T22:54:25.823 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.023+0000 7f4cc4cdf700 1 -- start start 2024-03-11T22:54:25.823 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.023+0000 7f4cc4cdf700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cc0102780 0x7f4cc0111590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:25.823 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.023+0000 7f4cc4cdf700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc0111ad0 con 0x7f4cc0102780 2024-03-11T22:54:25.823 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.023+0000 7f4cbe59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cc0102780 0x7f4cc0111590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:25.823 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.023+0000 7f4cbe59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cc0102780 0x7f4cc0111590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52644/0 (socket says 172.21.15.153:52644) 2024-03-11T22:54:25.823 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.023+0000 7f4cbe59c700 1 -- 172.21.15.153:0/1743658402 learned_addr learned my addr 172.21.15.153:0/1743658402 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:54:25.823 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.023+0000 7f4cbe59c700 1 -- 172.21.15.153:0/1743658402 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb00094f0 con 0x7f4cc0102780 2024-03-11T22:54:25.824 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.023+0000 7f4cbe59c700 1 --2- 172.21.15.153:0/1743658402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cc0102780 0x7f4cc0111590 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4cb00090b0 tx=0x7f4cb0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:25.824 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.023+0000 7f4caeffd700 1 -- 172.21.15.153:0/1743658402 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4cb0010840 con 0x7f4cc0102780 2024-03-11T22:54:25.824 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.023+0000 7f4caeffd700 1 -- 172.21.15.153:0/1743658402 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4cb0010e70 con 0x7f4cc0102780 2024-03-11T22:54:25.824 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.023+0000 7f4cc4cdf700 1 -- 172.21.15.153:0/1743658402 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cc0111cd0 con 0x7f4cc0102780 2024-03-11T22:54:25.824 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.023+0000 7f4caeffd700 1 -- 172.21.15.153:0/1743658402 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4cb0024e40 con 0x7f4cc0102780 2024-03-11T22:54:25.824 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.023+0000 7f4cc4cdf700 1 -- 172.21.15.153:0/1743658402 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cc0112970 con 0x7f4cc0102780 2024-03-11T22:54:25.824 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.024+0000 7f4caeffd700 1 -- 172.21.15.153:0/1743658402 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f4cb0024430 con 0x7f4cc0102780 2024-03-11T22:54:25.824 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.024+0000 7f4cc4cdf700 1 -- 172.21.15.153:0/1743658402 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cc005f5c0 con 0x7f4cc0102780 2024-03-11T22:54:25.824 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.024+0000 7f4caeffd700 1 --2- 172.21.15.153:0/1743658402 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f4ca8036880 0x7f4ca8038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:25.824 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.025+0000 7f4caeffd700 1 -- 172.21.15.153:0/1743658402 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4cb004a9a0 con 0x7f4cc0102780 2024-03-11T22:54:25.824 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.025+0000 7f4cbdd9b700 1 --2- 172.21.15.153:0/1743658402 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f4ca8036880 0x7f4ca8038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:25.824 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.025+0000 7f4cbdd9b700 1 --2- 172.21.15.153:0/1743658402 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f4ca8036880 0x7f4ca8038d30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4009990 tx=0x7f4cb4006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:25.824 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.029+0000 7f4caeffd700 1 -- 172.21.15.153:0/1743658402 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f4cb002c930 con 0x7f4cc0102780 2024-03-11T22:54:25.824 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.317+0000 7f4cc4cdf700 1 -- 172.21.15.153:0/1743658402 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f4cc004bed0 con 0x7f4cc0102780 2024-03-11T22:54:25.824 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.322+0000 7f4caeffd700 1 -- 172.21.15.153:0/1743658402 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7f4cb002c340 con 0x7f4cc0102780 2024-03-11T22:54:25.825 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.324+0000 7f4cc4cdf700 1 -- 172.21.15.153:0/1743658402 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f4ca8036880 msgr2=0x7f4ca8038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:25.825 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.324+0000 7f4cc4cdf700 1 --2- 172.21.15.153:0/1743658402 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f4ca8036880 0x7f4ca8038d30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4cb4009990 tx=0x7f4cb4006e40 comp rx=0 tx=0).stop 2024-03-11T22:54:25.825 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.324+0000 7f4cc4cdf700 1 -- 172.21.15.153:0/1743658402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cc0102780 msgr2=0x7f4cc0111590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:25.825 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.324+0000 7f4cc4cdf700 1 --2- 172.21.15.153:0/1743658402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cc0102780 0x7f4cc0111590 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4cb00090b0 tx=0x7f4cb0009250 comp rx=0 tx=0).stop 2024-03-11T22:54:25.825 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.324+0000 7f4cc4cdf700 1 -- 172.21.15.153:0/1743658402 shutdown_connections 2024-03-11T22:54:25.825 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.324+0000 7f4cc4cdf700 1 --2- 172.21.15.153:0/1743658402 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f4ca8036880 0x7f4ca8038d30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:25.825 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.324+0000 7f4cc4cdf700 1 --2- 172.21.15.153:0/1743658402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4cc0102780 0x7f4cc0111590 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:25.825 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.324+0000 7f4cc4cdf700 1 -- 172.21.15.153:0/1743658402 >> 172.21.15.153:0/1743658402 conn(0x7f4cc00fdf00 msgr2=0x7f4cc00fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:25.825 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.324+0000 7f4cc4cdf700 1 -- 172.21.15.153:0/1743658402 shutdown_connections 2024-03-11T22:54:25.825 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-03-11T22:54:25.324+0000 7f4cc4cdf700 1 -- 172.21.15.153:0/1743658402 wait complete. 2024-03-11T22:54:25.825 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-11T22:54:25.825 INFO:teuthology.orchestra.run.smithi153.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-11T22:54:25.825 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:54:25.825 INFO:teuthology.orchestra.run.smithi153.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:54:25.826 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:54:25.826 INFO:teuthology.orchestra.run.smithi153.stdout:Or, if you are only running a single cluster on this host: 2024-03-11T22:54:25.826 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:54:25.826 INFO:teuthology.orchestra.run.smithi153.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-11T22:54:25.826 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:54:25.826 INFO:teuthology.orchestra.run.smithi153.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-11T22:54:25.826 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:54:25.826 INFO:teuthology.orchestra.run.smithi153.stdout: ceph telemetry on 2024-03-11T22:54:25.826 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:54:25.826 INFO:teuthology.orchestra.run.smithi153.stdout:For more information see: 2024-03-11T22:54:25.826 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:54:25.826 INFO:teuthology.orchestra.run.smithi153.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-03-11T22:54:25.826 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:54:25.826 INFO:teuthology.orchestra.run.smithi153.stdout:Bootstrap complete. 2024-03-11T22:54:25.856 INFO:tasks.cephadm:Fetching config... 2024-03-11T22:54:25.856 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-11T22:54:25.856 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-11T22:54:25.871 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-11T22:54:25.871 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-11T22:54:25.871 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-11T22:54:25.925 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-11T22:54:25.925 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-11T22:54:25.925 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/keyring of=/dev/stdout 2024-03-11T22:54:25.992 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-11T22:54:25.992 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-11T22:54:25.992 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-11T22:54:26.047 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-11T22:54:26.047 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDp0ij5R7OMV1MF1G+8CFK/xpnLxZaeuNfwTyKXu1PU+hiDbbA1Iru7iy4Yfufv1sNk6XLzjZ+/oArCGtNNWb9xJUqpv4eHknSqYx1FBIYn7ibvkVwODY+oO31NuGDYrXPBJxGSKiQu+g6ysVhFOvza94GxuiSYa7DeXX0SAQL74CeTrrVgJNCi+Nacd9Gg+arTKUhkQzhOhX0EnOa7LkMulN83oSi+2Ba+F/mfDDKuUA3THOb0RSln/96G2bFV8w7kfTfPzPQzKQXMmv/MJjlOG//vPibUWD7cngYI7eGKrH+HCkg8TsFq/823h2gfmCBFkCkzesRRZHnOGwGdPnLXKOKAkSxFmkMun02BVEuUhEgimADNbrKQilPX9k0mHBnpl2zQHF0PI40nxjr1YrbuD1luVqkziRZEEz6/ahkO6flunB/auAgF6iO2FqL1NAZFY564QGdij80sN2xhgP6DfMnYxG0AFcDQCZr4P2GACZoWh9t34Bcq7gnVi1AaUtk= ceph-fa913b96-dff9-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T22:54:26.129 INFO:teuthology.orchestra.run.smithi153.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDp0ij5R7OMV1MF1G+8CFK/xpnLxZaeuNfwTyKXu1PU+hiDbbA1Iru7iy4Yfufv1sNk6XLzjZ+/oArCGtNNWb9xJUqpv4eHknSqYx1FBIYn7ibvkVwODY+oO31NuGDYrXPBJxGSKiQu+g6ysVhFOvza94GxuiSYa7DeXX0SAQL74CeTrrVgJNCi+Nacd9Gg+arTKUhkQzhOhX0EnOa7LkMulN83oSi+2Ba+F/mfDDKuUA3THOb0RSln/96G2bFV8w7kfTfPzPQzKQXMmv/MJjlOG//vPibUWD7cngYI7eGKrH+HCkg8TsFq/823h2gfmCBFkCkzesRRZHnOGwGdPnLXKOKAkSxFmkMun02BVEuUhEgimADNbrKQilPX9k0mHBnpl2zQHF0PI40nxjr1YrbuD1luVqkziRZEEz6/ahkO6flunB/auAgF6iO2FqL1NAZFY564QGdij80sN2xhgP6DfMnYxG0AFcDQCZr4P2GACZoWh9t34Bcq7gnVi1AaUtk= ceph-fa913b96-dff9-11ee-95c8-87774f69a715 2024-03-11T22:54:26.146 DEBUG:teuthology.orchestra.run.smithi163:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDp0ij5R7OMV1MF1G+8CFK/xpnLxZaeuNfwTyKXu1PU+hiDbbA1Iru7iy4Yfufv1sNk6XLzjZ+/oArCGtNNWb9xJUqpv4eHknSqYx1FBIYn7ibvkVwODY+oO31NuGDYrXPBJxGSKiQu+g6ysVhFOvza94GxuiSYa7DeXX0SAQL74CeTrrVgJNCi+Nacd9Gg+arTKUhkQzhOhX0EnOa7LkMulN83oSi+2Ba+F/mfDDKuUA3THOb0RSln/96G2bFV8w7kfTfPzPQzKQXMmv/MJjlOG//vPibUWD7cngYI7eGKrH+HCkg8TsFq/823h2gfmCBFkCkzesRRZHnOGwGdPnLXKOKAkSxFmkMun02BVEuUhEgimADNbrKQilPX9k0mHBnpl2zQHF0PI40nxjr1YrbuD1luVqkziRZEEz6/ahkO6flunB/auAgF6iO2FqL1NAZFY564QGdij80sN2xhgP6DfMnYxG0AFcDQCZr4P2GACZoWh9t34Bcq7gnVi1AaUtk= ceph-fa913b96-dff9-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T22:54:26.192 INFO:teuthology.orchestra.run.smithi163.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDp0ij5R7OMV1MF1G+8CFK/xpnLxZaeuNfwTyKXu1PU+hiDbbA1Iru7iy4Yfufv1sNk6XLzjZ+/oArCGtNNWb9xJUqpv4eHknSqYx1FBIYn7ibvkVwODY+oO31NuGDYrXPBJxGSKiQu+g6ysVhFOvza94GxuiSYa7DeXX0SAQL74CeTrrVgJNCi+Nacd9Gg+arTKUhkQzhOhX0EnOa7LkMulN83oSi+2Ba+F/mfDDKuUA3THOb0RSln/96G2bFV8w7kfTfPzPQzKQXMmv/MJjlOG//vPibUWD7cngYI7eGKrH+HCkg8TsFq/823h2gfmCBFkCkzesRRZHnOGwGdPnLXKOKAkSxFmkMun02BVEuUhEgimADNbrKQilPX9k0mHBnpl2zQHF0PI40nxjr1YrbuD1luVqkziRZEEz6/ahkO6flunB/auAgF6iO2FqL1NAZFY564QGdij80sN2xhgP6DfMnYxG0AFcDQCZr4P2GACZoWh9t34Bcq7gnVi1AaUtk= ceph-fa913b96-dff9-11ee-95c8-87774f69a715 2024-03-11T22:54:26.207 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-11T22:54:26.464 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:54:26.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:26 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/1743658402' entity='client.admin' 2024-03-11T22:54:28.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:28.874+0000 7f9a3bde3700 1 -- 172.21.15.153:0/2469062871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a34065db0 msgr2=0x7f9a340ff7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:28.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:28.874+0000 7f9a3bde3700 1 --2- 172.21.15.153:0/2469062871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a34065db0 0x7f9a340ff7c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9a300048c0 tx=0x7f9a30009e30 comp rx=0 tx=0).stop 2024-03-11T22:54:28.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:28.874+0000 7f9a3bde3700 1 -- 172.21.15.153:0/2469062871 shutdown_connections 2024-03-11T22:54:28.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:28.874+0000 7f9a3bde3700 1 --2- 172.21.15.153:0/2469062871 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a34065db0 0x7f9a340ff7c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:28.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:28.874+0000 7f9a3bde3700 1 -- 172.21.15.153:0/2469062871 >> 172.21.15.153:0/2469062871 conn(0x7f9a340f7190 msgr2=0x7f9a340f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:28.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:28.874+0000 7f9a3bde3700 1 -- 172.21.15.153:0/2469062871 shutdown_connections 2024-03-11T22:54:28.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:28.874+0000 7f9a3bde3700 1 -- 172.21.15.153:0/2469062871 wait complete. 2024-03-11T22:54:28.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:28.874+0000 7f9a3bde3700 1 Processor -- start 2024-03-11T22:54:28.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:28.874+0000 7f9a3bde3700 1 -- start start 2024-03-11T22:54:28.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:28.875+0000 7f9a3bde3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a34065db0 0x7f9a3410ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:28.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:28.875+0000 7f9a3bde3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3410d3d0 con 0x7f9a34065db0 2024-03-11T22:54:28.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:28.875+0000 7f9a39b7f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a34065db0 0x7f9a3410ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:28.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:28.875+0000 7f9a39b7f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a34065db0 0x7f9a3410ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52652/0 (socket says 172.21.15.153:52652) 2024-03-11T22:54:28.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:28.875+0000 7f9a39b7f700 1 -- 172.21.15.153:0/1348870625 learned_addr learned my addr 172.21.15.153:0/1348870625 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:54:28.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:28.875+0000 7f9a39b7f700 1 -- 172.21.15.153:0/1348870625 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a30005040 con 0x7f9a34065db0 2024-03-11T22:54:28.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:28.875+0000 7f9a39b7f700 1 --2- 172.21.15.153:0/1348870625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a34065db0 0x7f9a3410ce90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9a30009680 tx=0x7f9a30009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:28.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:28.875+0000 7f9a2a7fc700 1 -- 172.21.15.153:0/1348870625 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a3000b5c0 con 0x7f9a34065db0 2024-03-11T22:54:28.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:28.875+0000 7f9a2a7fc700 1 -- 172.21.15.153:0/1348870625 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9a30004e60 con 0x7f9a34065db0 2024-03-11T22:54:28.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:28.875+0000 7f9a3bde3700 1 -- 172.21.15.153:0/1348870625 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a3410d5d0 con 0x7f9a34065db0 2024-03-11T22:54:28.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:28.875+0000 7f9a2a7fc700 1 -- 172.21.15.153:0/1348870625 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a3001c590 con 0x7f9a34065db0 2024-03-11T22:54:28.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:28.876+0000 7f9a3bde3700 1 -- 172.21.15.153:0/1348870625 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a3410e270 con 0x7f9a34065db0 2024-03-11T22:54:28.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:28.877+0000 7f9a2a7fc700 1 -- 172.21.15.153:0/1348870625 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f9a3001c7c0 con 0x7f9a34065db0 2024-03-11T22:54:28.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:28.877+0000 7f9a3bde3700 1 -- 172.21.15.153:0/1348870625 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a3405f5c0 con 0x7f9a34065db0 2024-03-11T22:54:28.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:28.877+0000 7f9a2a7fc700 1 --2- 172.21.15.153:0/1348870625 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f9a200368c0 0x7f9a20038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:28.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:28.877+0000 7f9a2a7fc700 1 -- 172.21.15.153:0/1348870625 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f9a30049a80 con 0x7f9a34065db0 2024-03-11T22:54:28.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:28.877+0000 7f9a3937e700 1 --2- 172.21.15.153:0/1348870625 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f9a200368c0 0x7f9a20038d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:28.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:28.877+0000 7f9a3937e700 1 --2- 172.21.15.153:0/1348870625 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f9a200368c0 0x7f9a20038d70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9a24009990 tx=0x7f9a24006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:28.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:28.881+0000 7f9a2a7fc700 1 -- 172.21.15.153:0/1348870625 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f9a30023300 con 0x7f9a34065db0 2024-03-11T22:54:29.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:29.090+0000 7f9a3bde3700 1 -- 172.21.15.153:0/1348870625 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f9a3410ee50 con 0x7f9a34065db0 2024-03-11T22:54:29.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:29.094+0000 7f9a2a7fc700 1 -- 172.21.15.153:0/1348870625 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f9a30011b60 con 0x7f9a34065db0 2024-03-11T22:54:29.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:29.096+0000 7f9a3bde3700 1 -- 172.21.15.153:0/1348870625 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f9a200368c0 msgr2=0x7f9a20038d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:29.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:29.096+0000 7f9a3bde3700 1 --2- 172.21.15.153:0/1348870625 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f9a200368c0 0x7f9a20038d70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9a24009990 tx=0x7f9a24006e40 comp rx=0 tx=0).stop 2024-03-11T22:54:29.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:29.096+0000 7f9a3bde3700 1 -- 172.21.15.153:0/1348870625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a34065db0 msgr2=0x7f9a3410ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:29.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:29.096+0000 7f9a3bde3700 1 --2- 172.21.15.153:0/1348870625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a34065db0 0x7f9a3410ce90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9a30009680 tx=0x7f9a30009e30 comp rx=0 tx=0).stop 2024-03-11T22:54:29.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:29.097+0000 7f9a3bde3700 1 -- 172.21.15.153:0/1348870625 shutdown_connections 2024-03-11T22:54:29.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:29.097+0000 7f9a3bde3700 1 --2- 172.21.15.153:0/1348870625 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f9a200368c0 0x7f9a20038d70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:29.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:29.097+0000 7f9a3bde3700 1 --2- 172.21.15.153:0/1348870625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a34065db0 0x7f9a3410ce90 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:29.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:29.097+0000 7f9a3bde3700 1 -- 172.21.15.153:0/1348870625 >> 172.21.15.153:0/1348870625 conn(0x7f9a340f7190 msgr2=0x7f9a340f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:29.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:29.097+0000 7f9a3bde3700 1 -- 172.21.15.153:0/1348870625 shutdown_connections 2024-03-11T22:54:29.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:29.097+0000 7f9a3bde3700 1 -- 172.21.15.153:0/1348870625 wait complete. 2024-03-11T22:54:30.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:30 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/1348870625' entity='client.admin' 2024-03-11T22:54:30.254 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:30 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:30.254 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:30 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:30.254 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:30 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config rm", "who": "osd/host:smithi153", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:54:30.254 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:30 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:30.254 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:30 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi153", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-11T22:54:30.254 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:30 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi153", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-11T22:54:30.254 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:30 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:54:30.255 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-11T22:54:30.255 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-11T22:54:31.391 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:54:31.454 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:31 smithi153 ceph-mon[28962]: Deploying daemon crash.smithi153 on smithi153 2024-03-11T22:54:33.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.255+0000 7fda6bc3a700 1 -- 172.21.15.153:0/1128946288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda640fbab0 msgr2=0x7fda640fbec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:33.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.255+0000 7fda6bc3a700 1 --2- 172.21.15.153:0/1128946288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda640fbab0 0x7fda640fbec0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fda600048c0 tx=0x7fda60009e30 comp rx=0 tx=0).stop 2024-03-11T22:54:33.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.255+0000 7fda6bc3a700 1 -- 172.21.15.153:0/1128946288 shutdown_connections 2024-03-11T22:54:33.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.255+0000 7fda6bc3a700 1 --2- 172.21.15.153:0/1128946288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda640fbab0 0x7fda640fbec0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:33.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.255+0000 7fda6bc3a700 1 -- 172.21.15.153:0/1128946288 >> 172.21.15.153:0/1128946288 conn(0x7fda640f7210 msgr2=0x7fda640f9640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:33.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.255+0000 7fda6bc3a700 1 -- 172.21.15.153:0/1128946288 shutdown_connections 2024-03-11T22:54:33.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.255+0000 7fda6bc3a700 1 -- 172.21.15.153:0/1128946288 wait complete. 2024-03-11T22:54:33.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.256+0000 7fda6bc3a700 1 Processor -- start 2024-03-11T22:54:33.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.256+0000 7fda6bc3a700 1 -- start start 2024-03-11T22:54:33.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.256+0000 7fda6bc3a700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda640fbab0 0x7fda6410be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:33.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.256+0000 7fda6bc3a700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda6410c350 con 0x7fda640fbab0 2024-03-11T22:54:33.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.256+0000 7fda699d6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda640fbab0 0x7fda6410be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:33.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.256+0000 7fda699d6700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda640fbab0 0x7fda6410be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44838/0 (socket says 172.21.15.153:44838) 2024-03-11T22:54:33.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.256+0000 7fda699d6700 1 -- 172.21.15.153:0/2424667004 learned_addr learned my addr 172.21.15.153:0/2424667004 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:54:33.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.256+0000 7fda699d6700 1 -- 172.21.15.153:0/2424667004 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda60005040 con 0x7fda640fbab0 2024-03-11T22:54:33.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.257+0000 7fda699d6700 1 --2- 172.21.15.153:0/2424667004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda640fbab0 0x7fda6410be10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fda60009680 tx=0x7fda60009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:33.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.257+0000 7fda5a7fc700 1 -- 172.21.15.153:0/2424667004 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda6000b5c0 con 0x7fda640fbab0 2024-03-11T22:54:33.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.257+0000 7fda5a7fc700 1 -- 172.21.15.153:0/2424667004 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fda60004e60 con 0x7fda640fbab0 2024-03-11T22:54:33.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.257+0000 7fda6bc3a700 1 -- 172.21.15.153:0/2424667004 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda6410c550 con 0x7fda640fbab0 2024-03-11T22:54:33.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.258+0000 7fda5a7fc700 1 -- 172.21.15.153:0/2424667004 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda6001c590 con 0x7fda640fbab0 2024-03-11T22:54:33.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.258+0000 7fda6bc3a700 1 -- 172.21.15.153:0/2424667004 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda6410d110 con 0x7fda640fbab0 2024-03-11T22:54:33.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.258+0000 7fda5a7fc700 1 -- 172.21.15.153:0/2424667004 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7fda6001c830 con 0x7fda640fbab0 2024-03-11T22:54:33.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.258+0000 7fda5a7fc700 1 --2- 172.21.15.153:0/2424667004 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7fda50036870 0x7fda50038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:33.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.258+0000 7fda5a7fc700 1 -- 172.21.15.153:0/2424667004 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fda60013a80 con 0x7fda640fbab0 2024-03-11T22:54:33.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.258+0000 7fda691d5700 1 --2- 172.21.15.153:0/2424667004 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7fda50036870 0x7fda50038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:33.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.259+0000 7fda6bc3a700 1 -- 172.21.15.153:0/2424667004 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda6404bed0 con 0x7fda640fbab0 2024-03-11T22:54:33.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.259+0000 7fda691d5700 1 --2- 172.21.15.153:0/2424667004 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7fda50036870 0x7fda50038d20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fda54009990 tx=0x7fda54006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:33.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.263+0000 7fda5a7fc700 1 -- 172.21.15.153:0/2424667004 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7fda60023300 con 0x7fda640fbab0 2024-03-11T22:54:33.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.480+0000 7fda6bc3a700 1 -- 172.21.15.153:0/2424667004 --> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fda6410dcd0 con 0x7fda50036870 2024-03-11T22:54:33.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.484+0000 7fda5a7fc700 1 -- 172.21.15.153:0/2424667004 <== mgr.14160 v2:172.21.15.153:6800/4212724556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fda6410dcd0 con 0x7fda50036870 2024-03-11T22:54:33.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.486+0000 7fda6bc3a700 1 -- 172.21.15.153:0/2424667004 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7fda50036870 msgr2=0x7fda50038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:33.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.486+0000 7fda6bc3a700 1 --2- 172.21.15.153:0/2424667004 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7fda50036870 0x7fda50038d20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fda54009990 tx=0x7fda54006e40 comp rx=0 tx=0).stop 2024-03-11T22:54:33.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.486+0000 7fda6bc3a700 1 -- 172.21.15.153:0/2424667004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda640fbab0 msgr2=0x7fda6410be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:33.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.486+0000 7fda6bc3a700 1 --2- 172.21.15.153:0/2424667004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda640fbab0 0x7fda6410be10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fda60009680 tx=0x7fda60009e30 comp rx=0 tx=0).stop 2024-03-11T22:54:33.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.486+0000 7fda6bc3a700 1 -- 172.21.15.153:0/2424667004 shutdown_connections 2024-03-11T22:54:33.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.486+0000 7fda6bc3a700 1 --2- 172.21.15.153:0/2424667004 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7fda50036870 0x7fda50038d20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:33.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.486+0000 7fda6bc3a700 1 --2- 172.21.15.153:0/2424667004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda640fbab0 0x7fda6410be10 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:33.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.486+0000 7fda6bc3a700 1 -- 172.21.15.153:0/2424667004 >> 172.21.15.153:0/2424667004 conn(0x7fda640f7210 msgr2=0x7fda640f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:33.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.486+0000 7fda6bc3a700 1 -- 172.21.15.153:0/2424667004 shutdown_connections 2024-03-11T22:54:33.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:33.486+0000 7fda6bc3a700 1 -- 172.21.15.153:0/2424667004 wait complete. 2024-03-11T22:54:34.164 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:34 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:34.164 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:34 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:34.164 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:34 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:34.164 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:34 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:34.165 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:34 smithi153 ceph-mon[28962]: Deploying daemon node-exporter.smithi153 on smithi153 2024-03-11T22:54:34.165 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:34 smithi153 ceph-mon[28962]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:54:34.165 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:34 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:34.188 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi163 2024-03-11T22:54:34.188 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-11T22:54:34.189 DEBUG:teuthology.orchestra.run.smithi163:> dd of=/etc/ceph/ceph.conf 2024-03-11T22:54:34.208 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-11T22:54:34.208 DEBUG:teuthology.orchestra.run.smithi163:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:54:34.265 INFO:tasks.cephadm:Adding host smithi163 to orchestrator... 2024-03-11T22:54:34.266 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph orch host add smithi163 2024-03-11T22:54:34.530 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:54:37.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:37.586+0000 7f5fd340b700 1 -- 172.21.15.153:0/4157750192 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fcc066660 msgr2=0x7f5fcc0ff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:37.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:37.586+0000 7f5fd340b700 1 --2- 172.21.15.153:0/4157750192 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fcc066660 0x7f5fcc0ff790 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc0048c0 tx=0x7f5fbc009e30 comp rx=0 tx=0).stop 2024-03-11T22:54:37.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:37.586+0000 7f5fd340b700 1 -- 172.21.15.153:0/4157750192 shutdown_connections 2024-03-11T22:54:37.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:37.586+0000 7f5fd340b700 1 --2- 172.21.15.153:0/4157750192 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fcc066660 0x7f5fcc0ff790 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:37.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:37.586+0000 7f5fd340b700 1 -- 172.21.15.153:0/4157750192 >> 172.21.15.153:0/4157750192 conn(0x7f5fcc0f71a0 msgr2=0x7f5fcc0f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:37.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:37.586+0000 7f5fd340b700 1 -- 172.21.15.153:0/4157750192 shutdown_connections 2024-03-11T22:54:37.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:37.586+0000 7f5fd340b700 1 -- 172.21.15.153:0/4157750192 wait complete. 2024-03-11T22:54:37.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:37.587+0000 7f5fd340b700 1 Processor -- start 2024-03-11T22:54:37.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:37.587+0000 7f5fd340b700 1 -- start start 2024-03-11T22:54:37.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:37.587+0000 7f5fd340b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fcc066660 0x7f5fcc0fd210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:37.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:37.587+0000 7f5fd340b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fcc0fb690 con 0x7f5fcc066660 2024-03-11T22:54:37.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:37.587+0000 7f5fd11a7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fcc066660 0x7f5fcc0fd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:37.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:37.587+0000 7f5fd11a7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fcc066660 0x7f5fcc0fd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44862/0 (socket says 172.21.15.153:44862) 2024-03-11T22:54:37.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:37.587+0000 7f5fd11a7700 1 -- 172.21.15.153:0/736449684 learned_addr learned my addr 172.21.15.153:0/736449684 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:54:37.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:37.588+0000 7f5fd11a7700 1 -- 172.21.15.153:0/736449684 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fbc005040 con 0x7f5fcc066660 2024-03-11T22:54:37.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:37.588+0000 7f5fd11a7700 1 --2- 172.21.15.153:0/736449684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fcc066660 0x7f5fcc0fd210 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc009680 tx=0x7f5fbc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:37.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:37.588+0000 7f5fc1ffb700 1 -- 172.21.15.153:0/736449684 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fbc00b5c0 con 0x7f5fcc066660 2024-03-11T22:54:37.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:37.588+0000 7f5fc1ffb700 1 -- 172.21.15.153:0/736449684 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5fbc004e60 con 0x7f5fcc066660 2024-03-11T22:54:37.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:37.588+0000 7f5fd340b700 1 -- 172.21.15.153:0/736449684 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fcc0fb890 con 0x7f5fcc066660 2024-03-11T22:54:37.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:37.588+0000 7f5fc1ffb700 1 -- 172.21.15.153:0/736449684 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fbc01c590 con 0x7f5fcc066660 2024-03-11T22:54:37.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:37.588+0000 7f5fd340b700 1 -- 172.21.15.153:0/736449684 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fcc0fbd80 con 0x7f5fcc066660 2024-03-11T22:54:37.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:37.589+0000 7f5fc1ffb700 1 -- 172.21.15.153:0/736449684 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f5fbc01c810 con 0x7f5fcc066660 2024-03-11T22:54:37.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:37.589+0000 7f5fd340b700 1 -- 172.21.15.153:0/736449684 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fcc106980 con 0x7f5fcc066660 2024-03-11T22:54:37.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:37.589+0000 7f5fc1ffb700 1 --2- 172.21.15.153:0/736449684 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f5fb8036870 0x7f5fb8038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:37.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:37.589+0000 7f5fc1ffb700 1 -- 172.21.15.153:0/736449684 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5fbc049ae0 con 0x7f5fcc066660 2024-03-11T22:54:37.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:37.589+0000 7f5fd09a6700 1 --2- 172.21.15.153:0/736449684 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f5fb8036870 0x7f5fb8038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:37.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:37.590+0000 7f5fd09a6700 1 --2- 172.21.15.153:0/736449684 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f5fb8036870 0x7f5fb8038d20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5fc8009990 tx=0x7f5fc8006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:37.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:37.594+0000 7f5fc1ffb700 1 -- 172.21.15.153:0/736449684 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f5fbc023300 con 0x7f5fcc066660 2024-03-11T22:54:37.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:37.815+0000 7f5fd340b700 1 -- 172.21.15.153:0/736449684 --> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi163", "target": ["mon-mgr", ""]}) v1 -- 0x7f5fcc02b310 con 0x7f5fb8036870 2024-03-11T22:54:38.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:38 smithi153 ceph-mon[28962]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:54:38.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:38 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:38.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:38 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:38.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:38 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:38.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:38 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:38.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:38 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:38.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:38 smithi153 ceph-mon[28962]: Deploying daemon alertmanager.smithi153 on smithi153 2024-03-11T22:54:39.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:39 smithi153 ceph-mon[28962]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi163", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:54:39.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:39 smithi153 ceph-mon[28962]: Deploying cephadm binary to smithi163 2024-03-11T22:54:39.954 INFO:teuthology.orchestra.run.smithi153.stdout:Added host 'smithi163' with addr '172.21.15.163' 2024-03-11T22:54:39.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:39.949+0000 7f5fc1ffb700 1 -- 172.21.15.153:0/736449684 <== mgr.14160 v2:172.21.15.153:6800/4212724556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f5fcc02b310 con 0x7f5fb8036870 2024-03-11T22:54:39.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:39.951+0000 7f5fd340b700 1 -- 172.21.15.153:0/736449684 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f5fb8036870 msgr2=0x7f5fb8038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:39.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:39.951+0000 7f5fd340b700 1 --2- 172.21.15.153:0/736449684 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f5fb8036870 0x7f5fb8038d20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5fc8009990 tx=0x7f5fc8006e40 comp rx=0 tx=0).stop 2024-03-11T22:54:39.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:39.952+0000 7f5fd340b700 1 -- 172.21.15.153:0/736449684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fcc066660 msgr2=0x7f5fcc0fd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:39.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:39.952+0000 7f5fd340b700 1 --2- 172.21.15.153:0/736449684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fcc066660 0x7f5fcc0fd210 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5fbc009680 tx=0x7f5fbc009e30 comp rx=0 tx=0).stop 2024-03-11T22:54:39.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:39.952+0000 7f5fd340b700 1 -- 172.21.15.153:0/736449684 shutdown_connections 2024-03-11T22:54:39.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:39.952+0000 7f5fd340b700 1 --2- 172.21.15.153:0/736449684 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f5fb8036870 0x7f5fb8038d20 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:39.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:39.952+0000 7f5fd340b700 1 --2- 172.21.15.153:0/736449684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fcc066660 0x7f5fcc0fd210 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:39.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:39.952+0000 7f5fd340b700 1 -- 172.21.15.153:0/736449684 >> 172.21.15.153:0/736449684 conn(0x7f5fcc0f71a0 msgr2=0x7f5fcc0f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:39.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:39.952+0000 7f5fd340b700 1 -- 172.21.15.153:0/736449684 shutdown_connections 2024-03-11T22:54:39.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:39.952+0000 7f5fd340b700 1 -- 172.21.15.153:0/736449684 wait complete. 2024-03-11T22:54:40.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:40 smithi153 ceph-mon[28962]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:54:40.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:40 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:41.514 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:41 smithi153 ceph-mon[28962]: Added host smithi163 2024-03-11T22:54:41.539 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-11T22:54:42.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:42 smithi153 ceph-mon[28962]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:54:42.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:42 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:43.959 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:54:44.307 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:44 smithi153 ceph-mon[28962]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:54:46.394 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:46 smithi153 ceph-mon[28962]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:54:46.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.707+0000 7f058e61c700 1 -- 172.21.15.153:0/122946428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05880fae80 msgr2=0x7f05880fb290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:46.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.707+0000 7f058e61c700 1 --2- 172.21.15.153:0/122946428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05880fae80 0x7f05880fb290 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f05780048c0 tx=0x7f0578009e30 comp rx=0 tx=0).stop 2024-03-11T22:54:46.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.707+0000 7f058e61c700 1 -- 172.21.15.153:0/122946428 shutdown_connections 2024-03-11T22:54:46.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.707+0000 7f058e61c700 1 --2- 172.21.15.153:0/122946428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05880fae80 0x7f05880fb290 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:46.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.707+0000 7f058e61c700 1 -- 172.21.15.153:0/122946428 >> 172.21.15.153:0/122946428 conn(0x7f05880f6920 msgr2=0x7f05880f8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:46.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.707+0000 7f058e61c700 1 -- 172.21.15.153:0/122946428 shutdown_connections 2024-03-11T22:54:46.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.707+0000 7f058e61c700 1 -- 172.21.15.153:0/122946428 wait complete. 2024-03-11T22:54:46.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.707+0000 7f058e61c700 1 Processor -- start 2024-03-11T22:54:46.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.708+0000 7f058e61c700 1 -- start start 2024-03-11T22:54:46.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.708+0000 7f058e61c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05880fae80 0x7f058810ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:46.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.708+0000 7f058e61c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058810d410 con 0x7f05880fae80 2024-03-11T22:54:46.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.708+0000 7f0587fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05880fae80 0x7f058810ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:46.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.708+0000 7f0587fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05880fae80 0x7f058810ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53758/0 (socket says 172.21.15.153:53758) 2024-03-11T22:54:46.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.708+0000 7f0587fff700 1 -- 172.21.15.153:0/3357895305 learned_addr learned my addr 172.21.15.153:0/3357895305 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:54:46.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.708+0000 7f0587fff700 1 -- 172.21.15.153:0/3357895305 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0578005040 con 0x7f05880fae80 2024-03-11T22:54:46.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.708+0000 7f0587fff700 1 --2- 172.21.15.153:0/3357895305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05880fae80 0x7f058810ced0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0578000c00 tx=0x7f0578009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:46.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.709+0000 7f0584ff9700 1 -- 172.21.15.153:0/3357895305 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f057800b5c0 con 0x7f05880fae80 2024-03-11T22:54:46.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.709+0000 7f058e61c700 1 -- 172.21.15.153:0/3357895305 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f058810d610 con 0x7f05880fae80 2024-03-11T22:54:46.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.709+0000 7f0584ff9700 1 -- 172.21.15.153:0/3357895305 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f057800be70 con 0x7f05880fae80 2024-03-11T22:54:46.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.709+0000 7f0584ff9700 1 -- 172.21.15.153:0/3357895305 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f057801c590 con 0x7f05880fae80 2024-03-11T22:54:46.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.709+0000 7f058e61c700 1 -- 172.21.15.153:0/3357895305 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f058810e1d0 con 0x7f05880fae80 2024-03-11T22:54:46.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.710+0000 7f0584ff9700 1 -- 172.21.15.153:0/3357895305 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f057801c850 con 0x7f05880fae80 2024-03-11T22:54:46.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.710+0000 7f0584ff9700 1 --2- 172.21.15.153:0/3357895305 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f0570036870 0x7f0570038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:46.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.710+0000 7f0584ff9700 1 -- 172.21.15.153:0/3357895305 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0578049a40 con 0x7f05880fae80 2024-03-11T22:54:46.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.710+0000 7f05877fe700 1 --2- 172.21.15.153:0/3357895305 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f0570036870 0x7f0570038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:46.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.710+0000 7f058e61c700 1 -- 172.21.15.153:0/3357895305 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f058805f5c0 con 0x7f05880fae80 2024-03-11T22:54:46.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.710+0000 7f05877fe700 1 --2- 172.21.15.153:0/3357895305 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f0570036870 0x7f0570038d20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f057c009990 tx=0x7f057c006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:46.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.715+0000 7f0584ff9700 1 -- 172.21.15.153:0/3357895305 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f0578023300 con 0x7f05880fae80 2024-03-11T22:54:46.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.938+0000 7f058e61c700 1 -- 172.21.15.153:0/3357895305 --> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f058810e4d0 con 0x7f0570036870 2024-03-11T22:54:46.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.940+0000 7f0584ff9700 1 -- 172.21.15.153:0/3357895305 <== mgr.14160 v2:172.21.15.153:6800/4212724556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7f058810e4d0 con 0x7f0570036870 2024-03-11T22:54:46.944 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:54:46.944 INFO:teuthology.orchestra.run.smithi153.stdout:[{"addr": "172.21.15.153", "hostname": "smithi153", "labels": [], "status": ""}, {"addr": "172.21.15.163", "hostname": "smithi163", "labels": [], "status": ""}] 2024-03-11T22:54:46.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.941+0000 7f058e61c700 1 -- 172.21.15.153:0/3357895305 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f0570036870 msgr2=0x7f0570038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:46.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.941+0000 7f058e61c700 1 --2- 172.21.15.153:0/3357895305 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f0570036870 0x7f0570038d20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f057c009990 tx=0x7f057c006e40 comp rx=0 tx=0).stop 2024-03-11T22:54:46.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.941+0000 7f058e61c700 1 -- 172.21.15.153:0/3357895305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05880fae80 msgr2=0x7f058810ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:46.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.941+0000 7f058e61c700 1 --2- 172.21.15.153:0/3357895305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05880fae80 0x7f058810ced0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0578000c00 tx=0x7f0578009e30 comp rx=0 tx=0).stop 2024-03-11T22:54:46.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.942+0000 7f058e61c700 1 -- 172.21.15.153:0/3357895305 shutdown_connections 2024-03-11T22:54:46.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.942+0000 7f058e61c700 1 --2- 172.21.15.153:0/3357895305 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f0570036870 0x7f0570038d20 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:46.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.942+0000 7f058e61c700 1 --2- 172.21.15.153:0/3357895305 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05880fae80 0x7f058810ced0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:46.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.942+0000 7f058e61c700 1 -- 172.21.15.153:0/3357895305 >> 172.21.15.153:0/3357895305 conn(0x7f05880f6920 msgr2=0x7f05880f7660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:46.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.942+0000 7f058e61c700 1 -- 172.21.15.153:0/3357895305 shutdown_connections 2024-03-11T22:54:46.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:46.942+0000 7f058e61c700 1 -- 172.21.15.153:0/3357895305 wait complete. 2024-03-11T22:54:47.510 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-11T22:54:47.510 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph osd crush tunables default 2024-03-11T22:54:47.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:47 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:47.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:47 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:47.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:47 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:47.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:47 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:47.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:47 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:47.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:47 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:47.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:47 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T22:54:47.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:47 smithi153 ceph-mon[28962]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T22:54:47.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:47 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:47.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:47 smithi153 ceph-mon[28962]: Deploying daemon grafana.smithi153 on smithi153 2024-03-11T22:54:47.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:47 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:47.768 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:54:48.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:48 smithi153 ceph-mon[28962]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:54:48.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:48 smithi153 ceph-mon[28962]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:54:49.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:49.568+0000 7fdadcc66700 1 -- 172.21.15.153:0/4143144370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdad80fd090 msgr2=0x7fdad80fd4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:49.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:49.568+0000 7fdadcc66700 1 --2- 172.21.15.153:0/4143144370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdad80fd090 0x7fdad80fd4a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fdac80048c0 tx=0x7fdac8009e30 comp rx=0 tx=0).stop 2024-03-11T22:54:49.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:49.569+0000 7fdadcc66700 1 -- 172.21.15.153:0/4143144370 shutdown_connections 2024-03-11T22:54:49.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:49.569+0000 7fdadcc66700 1 --2- 172.21.15.153:0/4143144370 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdad80fd090 0x7fdad80fd4a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:49.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:49.569+0000 7fdadcc66700 1 -- 172.21.15.153:0/4143144370 >> 172.21.15.153:0/4143144370 conn(0x7fdad80f6920 msgr2=0x7fdad80f8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:49.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:49.569+0000 7fdadcc66700 1 -- 172.21.15.153:0/4143144370 shutdown_connections 2024-03-11T22:54:49.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:49.569+0000 7fdadcc66700 1 -- 172.21.15.153:0/4143144370 wait complete. 2024-03-11T22:54:49.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:49.569+0000 7fdadcc66700 1 Processor -- start 2024-03-11T22:54:49.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:49.569+0000 7fdadcc66700 1 -- start start 2024-03-11T22:54:49.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:49.569+0000 7fdadcc66700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdad80fd090 0x7fdad810f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:49.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:49.569+0000 7fdadcc66700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad810f620 con 0x7fdad80fd090 2024-03-11T22:54:49.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:49.569+0000 7fdad659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdad80fd090 0x7fdad810f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:49.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:49.570+0000 7fdad659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdad80fd090 0x7fdad810f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53778/0 (socket says 172.21.15.153:53778) 2024-03-11T22:54:49.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:49.570+0000 7fdad659c700 1 -- 172.21.15.153:0/655306157 learned_addr learned my addr 172.21.15.153:0/655306157 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:54:49.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:49.570+0000 7fdad659c700 1 -- 172.21.15.153:0/655306157 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdac8005040 con 0x7fdad80fd090 2024-03-11T22:54:49.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:49.570+0000 7fdad659c700 1 --2- 172.21.15.153:0/655306157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdad80fd090 0x7fdad810f0e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fdac8009b80 tx=0x7fdac8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:49.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:49.570+0000 7fdac6ffd700 1 -- 172.21.15.153:0/655306157 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdac800b5c0 con 0x7fdad80fd090 2024-03-11T22:54:49.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:49.570+0000 7fdadcc66700 1 -- 172.21.15.153:0/655306157 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdad810f820 con 0x7fdad80fd090 2024-03-11T22:54:49.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:49.570+0000 7fdac6ffd700 1 -- 172.21.15.153:0/655306157 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdac800be70 con 0x7fdad80fd090 2024-03-11T22:54:49.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:49.570+0000 7fdac6ffd700 1 -- 172.21.15.153:0/655306157 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdac801c660 con 0x7fdad80fd090 2024-03-11T22:54:49.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:49.570+0000 7fdadcc66700 1 -- 172.21.15.153:0/655306157 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdad81104c0 con 0x7fdad80fd090 2024-03-11T22:54:49.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:49.571+0000 7fdac6ffd700 1 -- 172.21.15.153:0/655306157 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7fdac801c8d0 con 0x7fdad80fd090 2024-03-11T22:54:49.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:49.571+0000 7fdac6ffd700 1 --2- 172.21.15.153:0/655306157 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7fdac0036870 0x7fdac0038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:49.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:49.571+0000 7fdac6ffd700 1 -- 172.21.15.153:0/655306157 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fdac804aae0 con 0x7fdad80fd090 2024-03-11T22:54:49.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:49.571+0000 7fdad5d9b700 1 --2- 172.21.15.153:0/655306157 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7fdac0036870 0x7fdac0038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:49.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:49.572+0000 7fdad5d9b700 1 --2- 172.21.15.153:0/655306157 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7fdac0036870 0x7fdac0038d20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fdacc009990 tx=0x7fdacc006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:49.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:49.572+0000 7fdadcc66700 1 -- 172.21.15.153:0/655306157 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdad805f5c0 con 0x7fdad80fd090 2024-03-11T22:54:49.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:49.577+0000 7fdac6ffd700 1 -- 172.21.15.153:0/655306157 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7fdac8011dd0 con 0x7fdad80fd090 2024-03-11T22:54:49.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:49.780+0000 7fdadcc66700 1 -- 172.21.15.153:0/655306157 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fdad8110ed0 con 0x7fdad80fd090 2024-03-11T22:54:50.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:50.335+0000 7fdac6ffd700 1 -- 172.21.15.153:0/655306157 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fdac8011370 con 0x7fdad80fd090 2024-03-11T22:54:50.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:50.338+0000 7fdadcc66700 1 -- 172.21.15.153:0/655306157 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7fdac0036870 msgr2=0x7fdac0038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:50.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:50.338+0000 7fdadcc66700 1 --2- 172.21.15.153:0/655306157 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7fdac0036870 0x7fdac0038d20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fdacc009990 tx=0x7fdacc006e40 comp rx=0 tx=0).stop 2024-03-11T22:54:50.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:50.338+0000 7fdadcc66700 1 -- 172.21.15.153:0/655306157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdad80fd090 msgr2=0x7fdad810f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:50.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:50.338+0000 7fdadcc66700 1 --2- 172.21.15.153:0/655306157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdad80fd090 0x7fdad810f0e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fdac8009b80 tx=0x7fdac8009e30 comp rx=0 tx=0).stop 2024-03-11T22:54:50.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:50.338+0000 7fdadcc66700 1 -- 172.21.15.153:0/655306157 shutdown_connections 2024-03-11T22:54:50.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:50.338+0000 7fdadcc66700 1 --2- 172.21.15.153:0/655306157 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7fdac0036870 0x7fdac0038d20 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:50.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:50.338+0000 7fdadcc66700 1 --2- 172.21.15.153:0/655306157 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdad80fd090 0x7fdad810f0e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:50.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:50.338+0000 7fdadcc66700 1 -- 172.21.15.153:0/655306157 >> 172.21.15.153:0/655306157 conn(0x7fdad80f6920 msgr2=0x7fdad80fb500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:50.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:50.338+0000 7fdadcc66700 1 -- 172.21.15.153:0/655306157 shutdown_connections 2024-03-11T22:54:50.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:54:50.338+0000 7fdadcc66700 1 -- 172.21.15.153:0/655306157 wait complete. 2024-03-11T22:54:50.344 INFO:teuthology.orchestra.run.smithi153.stderr:adjusted tunables profile to default 2024-03-11T22:54:50.725 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:50 smithi153 ceph-mon[28962]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:54:50.725 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:50 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/655306157' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-11T22:54:51.023 INFO:tasks.cephadm:Adding mon.smithi153 on smithi153 2024-03-11T22:54:51.023 INFO:tasks.cephadm:Adding mon.smithi163 on smithi163 2024-03-11T22:54:51.024 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph orch apply mon '2;smithi153:172.21.15.153=smithi153;smithi163:172.21.15.163=smithi163' 2024-03-11T22:54:51.272 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-11T22:54:51.319 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-11T22:54:51.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:51 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/655306157' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-11T22:54:51.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:51 smithi153 ceph-mon[28962]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:54:52.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.701+0000 7f5744aff700 1 -- 172.21.15.163:0/1026004177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57400fbac0 msgr2=0x7f57400fbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:52.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.701+0000 7f5744aff700 1 --2- 172.21.15.163:0/1026004177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57400fbac0 0x7f57400fbed0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5730009c60 tx=0x7f5730004d10 comp rx=0 tx=0).stop 2024-03-11T22:54:52.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.701+0000 7f5744aff700 1 -- 172.21.15.163:0/1026004177 shutdown_connections 2024-03-11T22:54:52.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.701+0000 7f5744aff700 1 --2- 172.21.15.163:0/1026004177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57400fbac0 0x7f57400fbed0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:52.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.701+0000 7f5744aff700 1 -- 172.21.15.163:0/1026004177 >> 172.21.15.163:0/1026004177 conn(0x7f57400f71a0 msgr2=0x7f57400f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:52.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.701+0000 7f5744aff700 1 -- 172.21.15.163:0/1026004177 shutdown_connections 2024-03-11T22:54:52.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.701+0000 7f5744aff700 1 -- 172.21.15.163:0/1026004177 wait complete. 2024-03-11T22:54:52.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.702+0000 7f5744aff700 1 Processor -- start 2024-03-11T22:54:52.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.702+0000 7f5744aff700 1 -- start start 2024-03-11T22:54:52.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.702+0000 7f5744aff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57400fbac0 0x7f574010fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:52.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.702+0000 7f5744aff700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57400667f0 con 0x7f57400fbac0 2024-03-11T22:54:52.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.702+0000 7f573e59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57400fbac0 0x7f574010fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:52.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.702+0000 7f573e59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57400fbac0 0x7f574010fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.163:37438/0 (socket says 172.21.15.163:37438) 2024-03-11T22:54:52.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.702+0000 7f573e59c700 1 -- 172.21.15.163:0/385433971 learned_addr learned my addr 172.21.15.163:0/385433971 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-03-11T22:54:52.708 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.702+0000 7f573e59c700 1 -- 172.21.15.163:0/385433971 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5730009940 con 0x7f57400fbac0 2024-03-11T22:54:52.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.702+0000 7f573e59c700 1 --2- 172.21.15.163:0/385433971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57400fbac0 0x7f574010fc00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5730015040 tx=0x7f5730004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:52.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.702+0000 7f5736ffd700 1 -- 172.21.15.163:0/385433971 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5730007c80 con 0x7f57400fbac0 2024-03-11T22:54:52.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.703+0000 7f5736ffd700 1 -- 172.21.15.163:0/385433971 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f573001d910 con 0x7f57400fbac0 2024-03-11T22:54:52.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.703+0000 7f5744aff700 1 -- 172.21.15.163:0/385433971 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57400669f0 con 0x7f57400fbac0 2024-03-11T22:54:52.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.703+0000 7f5736ffd700 1 -- 172.21.15.163:0/385433971 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57300129f0 con 0x7f57400fbac0 2024-03-11T22:54:52.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.703+0000 7f5744aff700 1 -- 172.21.15.163:0/385433971 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5740110380 con 0x7f57400fbac0 2024-03-11T22:54:52.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.703+0000 7f5736ffd700 1 -- 172.21.15.163:0/385433971 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f5730019040 con 0x7f57400fbac0 2024-03-11T22:54:52.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.704+0000 7f5736ffd700 1 --2- 172.21.15.163:0/385433971 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f572c036870 0x7f572c038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:52.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.704+0000 7f5736ffd700 1 -- 172.21.15.163:0/385433971 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f573004a490 con 0x7f57400fbac0 2024-03-11T22:54:52.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.704+0000 7f5744aff700 1 -- 172.21.15.163:0/385433971 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5740105890 con 0x7f57400fbac0 2024-03-11T22:54:52.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.704+0000 7f573dd9b700 1 --2- 172.21.15.163:0/385433971 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f572c036870 0x7f572c038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:52.709 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.704+0000 7f573dd9b700 1 --2- 172.21.15.163:0/385433971 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f572c036870 0x7f572c038d20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5728009990 tx=0x7f5728006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:52.713 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.709+0000 7f5736ffd700 1 -- 172.21.15.163:0/385433971 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f5730021070 con 0x7f57400fbac0 2024-03-11T22:54:52.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:52 smithi153 ceph-mon[28962]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:54:52.904 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.900+0000 7f5744aff700 1 -- 172.21.15.163:0/385433971 --> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi153:172.21.15.153=smithi153;smithi163:172.21.15.163=smithi163", "target": ["mon-mgr", ""]}) v1 -- 0x7f574002b3c0 con 0x7f572c036870 2024-03-11T22:54:52.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.907+0000 7f5736ffd700 1 -- 172.21.15.163:0/385433971 <== mgr.14160 v2:172.21.15.153:6800/4212724556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f574002b3c0 con 0x7f572c036870 2024-03-11T22:54:52.911 INFO:teuthology.orchestra.run.smithi163.stdout:Scheduled mon update... 2024-03-11T22:54:52.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.908+0000 7f5744aff700 1 -- 172.21.15.163:0/385433971 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f572c036870 msgr2=0x7f572c038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:52.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.908+0000 7f5744aff700 1 --2- 172.21.15.163:0/385433971 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f572c036870 0x7f572c038d20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5728009990 tx=0x7f5728006e40 comp rx=0 tx=0).stop 2024-03-11T22:54:52.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.908+0000 7f5744aff700 1 -- 172.21.15.163:0/385433971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57400fbac0 msgr2=0x7f574010fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:52.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.908+0000 7f5744aff700 1 --2- 172.21.15.163:0/385433971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57400fbac0 0x7f574010fc00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5730015040 tx=0x7f5730004d10 comp rx=0 tx=0).stop 2024-03-11T22:54:52.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.908+0000 7f5744aff700 1 -- 172.21.15.163:0/385433971 shutdown_connections 2024-03-11T22:54:52.913 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.908+0000 7f5744aff700 1 --2- 172.21.15.163:0/385433971 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f572c036870 0x7f572c038d20 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:52.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.908+0000 7f5744aff700 1 --2- 172.21.15.163:0/385433971 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f57400fbac0 0x7f574010fc00 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:52.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.908+0000 7f5744aff700 1 -- 172.21.15.163:0/385433971 >> 172.21.15.163:0/385433971 conn(0x7f57400f71a0 msgr2=0x7f57400f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:52.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.909+0000 7f5744aff700 1 -- 172.21.15.163:0/385433971 shutdown_connections 2024-03-11T22:54:52.914 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:52.909+0000 7f5744aff700 1 -- 172.21.15.163:0/385433971 wait complete. 2024-03-11T22:54:53.469 DEBUG:teuthology.orchestra.run.smithi163:mon.smithi163> sudo journalctl -f -n 0 -u ceph-fa913b96-dff9-11ee-95c8-87774f69a715@mon.smithi163.service 2024-03-11T22:54:53.472 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-11T22:54:53.472 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:54:53.746 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-11T22:54:53.793 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-11T22:54:54.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:53 smithi153 ceph-mon[28962]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi153:172.21.15.153=smithi153;smithi163:172.21.15.163=smithi163", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:54:54.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:53 smithi153 ceph-mon[28962]: Saving service mon spec with placement smithi153:172.21.15.153=smithi153;smithi163:172.21.15.163=smithi163;count:2 2024-03-11T22:54:54.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:53 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:54:54.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:53 smithi153 ceph-mon[28962]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:54:55.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.230+0000 7f10d2896700 1 -- 172.21.15.163:0/267387987 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10cc0fae80 msgr2=0x7f10cc0fb290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:55.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.230+0000 7f10d2896700 1 --2- 172.21.15.163:0/267387987 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10cc0fae80 0x7f10cc0fb290 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f10bc0048c0 tx=0x7f10bc009e30 comp rx=0 tx=0).stop 2024-03-11T22:54:55.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.230+0000 7f10d2896700 1 -- 172.21.15.163:0/267387987 shutdown_connections 2024-03-11T22:54:55.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.230+0000 7f10d2896700 1 --2- 172.21.15.163:0/267387987 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10cc0fae80 0x7f10cc0fb290 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:55.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.230+0000 7f10d2896700 1 -- 172.21.15.163:0/267387987 >> 172.21.15.163:0/267387987 conn(0x7f10cc0f6920 msgr2=0x7f10cc0f8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:55.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.230+0000 7f10d2896700 1 -- 172.21.15.163:0/267387987 shutdown_connections 2024-03-11T22:54:55.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.230+0000 7f10d2896700 1 -- 172.21.15.163:0/267387987 wait complete. 2024-03-11T22:54:55.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.230+0000 7f10d2896700 1 Processor -- start 2024-03-11T22:54:55.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.230+0000 7f10d2896700 1 -- start start 2024-03-11T22:54:55.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.231+0000 7f10d2896700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10cc0fae80 0x7f10cc0fd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:55.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.231+0000 7f10d2896700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10cc0fbe20 con 0x7f10cc0fae80 2024-03-11T22:54:55.236 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.231+0000 7f10cbfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10cc0fae80 0x7f10cc0fd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:55.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.231+0000 7f10cbfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10cc0fae80 0x7f10cc0fd9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.163:37456/0 (socket says 172.21.15.163:37456) 2024-03-11T22:54:55.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.231+0000 7f10cbfff700 1 -- 172.21.15.163:0/2674362688 learned_addr learned my addr 172.21.15.163:0/2674362688 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-03-11T22:54:55.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.231+0000 7f10cbfff700 1 -- 172.21.15.163:0/2674362688 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10bc005040 con 0x7f10cc0fae80 2024-03-11T22:54:55.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.231+0000 7f10cbfff700 1 --2- 172.21.15.163:0/2674362688 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10cc0fae80 0x7f10cc0fd9a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f10bc009040 tx=0x7f10bc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:55.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.231+0000 7f10c8ff9700 1 -- 172.21.15.163:0/2674362688 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10bc00bed0 con 0x7f10cc0fae80 2024-03-11T22:54:55.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.231+0000 7f10c8ff9700 1 -- 172.21.15.163:0/2674362688 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f10bc00b5c0 con 0x7f10cc0fae80 2024-03-11T22:54:55.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.231+0000 7f10c8ff9700 1 -- 172.21.15.163:0/2674362688 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f10bc01c5b0 con 0x7f10cc0fae80 2024-03-11T22:54:55.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.231+0000 7f10d2896700 1 -- 172.21.15.163:0/2674362688 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10cc0fc020 con 0x7f10cc0fae80 2024-03-11T22:54:55.237 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.231+0000 7f10d2896700 1 -- 172.21.15.163:0/2674362688 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10cc0fc510 con 0x7f10cc0fae80 2024-03-11T22:54:55.238 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.232+0000 7f10c8ff9700 1 -- 172.21.15.163:0/2674362688 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f10bc013420 con 0x7f10cc0fae80 2024-03-11T22:54:55.238 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.232+0000 7f10d2896700 1 -- 172.21.15.163:0/2674362688 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10cc106930 con 0x7f10cc0fae80 2024-03-11T22:54:55.238 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.232+0000 7f10c8ff9700 1 --2- 172.21.15.163:0/2674362688 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f10b4036870 0x7f10b4038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:55.238 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.232+0000 7f10c8ff9700 1 -- 172.21.15.163:0/2674362688 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f10bc049b80 con 0x7f10cc0fae80 2024-03-11T22:54:55.238 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.233+0000 7f10cb7fe700 1 --2- 172.21.15.163:0/2674362688 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f10b4036870 0x7f10b4038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:55.238 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.233+0000 7f10cb7fe700 1 --2- 172.21.15.163:0/2674362688 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f10b4036870 0x7f10b4038d20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f10c0009990 tx=0x7f10c0006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:55.243 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.239+0000 7f10c8ff9700 1 -- 172.21.15.163:0/2674362688 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f10bc04be60 con 0x7f10cc0fae80 2024-03-11T22:54:55.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.494+0000 7f10d2896700 1 -- 172.21.15.163:0/2674362688 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f10cc04bed0 con 0x7f10cc0fae80 2024-03-11T22:54:55.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.495+0000 7f10c8ff9700 1 -- 172.21.15.163:0/2674362688 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f10bc013a80 con 0x7f10cc0fae80 2024-03-11T22:54:55.499 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:54:55.499 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":1,"fsid":"fa913b96-dff9-11ee-95c8-87774f69a715","modified":"2024-03-11T22:53:11.271993Z","created":"2024-03-11T22:53:11.271993Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T22:54:55.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.497+0000 7f10d2896700 1 -- 172.21.15.163:0/2674362688 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f10b4036870 msgr2=0x7f10b4038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:55.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.497+0000 7f10d2896700 1 --2- 172.21.15.163:0/2674362688 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f10b4036870 0x7f10b4038d20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f10c0009990 tx=0x7f10c0006e40 comp rx=0 tx=0).stop 2024-03-11T22:54:55.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.497+0000 7f10d2896700 1 -- 172.21.15.163:0/2674362688 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10cc0fae80 msgr2=0x7f10cc0fd9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:55.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.497+0000 7f10d2896700 1 --2- 172.21.15.163:0/2674362688 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10cc0fae80 0x7f10cc0fd9a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f10bc009040 tx=0x7f10bc009e30 comp rx=0 tx=0).stop 2024-03-11T22:54:55.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.497+0000 7f10d2896700 1 -- 172.21.15.163:0/2674362688 shutdown_connections 2024-03-11T22:54:55.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.497+0000 7f10d2896700 1 --2- 172.21.15.163:0/2674362688 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f10b4036870 0x7f10b4038d20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:55.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.497+0000 7f10d2896700 1 --2- 172.21.15.163:0/2674362688 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10cc0fae80 0x7f10cc0fd9a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:55.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.497+0000 7f10d2896700 1 -- 172.21.15.163:0/2674362688 >> 172.21.15.163:0/2674362688 conn(0x7f10cc0f6920 msgr2=0x7f10cc0f7640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:55.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.497+0000 7f10d2896700 1 -- 172.21.15.163:0/2674362688 shutdown_connections 2024-03-11T22:54:55.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:55.497+0000 7f10d2896700 1 -- 172.21.15.163:0/2674362688 wait complete. 2024-03-11T22:54:55.502 INFO:teuthology.orchestra.run.smithi163.stderr:dumped monmap epoch 1 2024-03-11T22:54:56.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:56 smithi153 ceph-mon[28962]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:54:56.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:56 smithi153 ceph-mon[28962]: from='client.? 172.21.15.163:0/2674362688' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:54:57.166 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-11T22:54:57.167 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:54:57.425 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-11T22:54:57.474 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-11T22:54:58.388 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:58 smithi153 ceph-mon[28962]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:54:58.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:58.903+0000 7f09b09ac700 1 -- 172.21.15.163:0/305942182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09ac0fbac0 msgr2=0x7f09ac0fbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:58.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:58.903+0000 7f09b09ac700 1 --2- 172.21.15.163:0/305942182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09ac0fbac0 0x7f09ac0fbed0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f09a00048c0 tx=0x7f09a0009e30 comp rx=0 tx=0).stop 2024-03-11T22:54:58.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:58.903+0000 7f09b09ac700 1 -- 172.21.15.163:0/305942182 shutdown_connections 2024-03-11T22:54:58.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:58.903+0000 7f09b09ac700 1 --2- 172.21.15.163:0/305942182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09ac0fbac0 0x7f09ac0fbed0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:58.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:58.903+0000 7f09b09ac700 1 -- 172.21.15.163:0/305942182 >> 172.21.15.163:0/305942182 conn(0x7f09ac0f71a0 msgr2=0x7f09ac0f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:58.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:58.903+0000 7f09b09ac700 1 -- 172.21.15.163:0/305942182 shutdown_connections 2024-03-11T22:54:58.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:58.903+0000 7f09b09ac700 1 -- 172.21.15.163:0/305942182 wait complete. 2024-03-11T22:54:58.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:58.903+0000 7f09b09ac700 1 Processor -- start 2024-03-11T22:54:58.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:58.903+0000 7f09b09ac700 1 -- start start 2024-03-11T22:54:58.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:58.903+0000 7f09b09ac700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09ac0fbac0 0x7f09ac10fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:58.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:58.903+0000 7f09b09ac700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ac0667f0 con 0x7f09ac0fbac0 2024-03-11T22:54:58.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:58.904+0000 7f09aa59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09ac0fbac0 0x7f09ac10fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:58.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:58.904+0000 7f09aa59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09ac0fbac0 0x7f09ac10fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.163:43858/0 (socket says 172.21.15.163:43858) 2024-03-11T22:54:58.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:58.904+0000 7f09aa59c700 1 -- 172.21.15.163:0/3155790713 learned_addr learned my addr 172.21.15.163:0/3155790713 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-03-11T22:54:58.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:58.904+0000 7f09aa59c700 1 -- 172.21.15.163:0/3155790713 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a0005040 con 0x7f09ac0fbac0 2024-03-11T22:54:58.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:58.904+0000 7f09aa59c700 1 --2- 172.21.15.163:0/3155790713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09ac0fbac0 0x7f09ac10fbf0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f09a0009680 tx=0x7f09a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:58.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:58.904+0000 7f099affd700 1 -- 172.21.15.163:0/3155790713 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09a000bed0 con 0x7f09ac0fbac0 2024-03-11T22:54:58.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:58.904+0000 7f099affd700 1 -- 172.21.15.163:0/3155790713 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f09a000b5c0 con 0x7f09ac0fbac0 2024-03-11T22:54:58.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:58.904+0000 7f099affd700 1 -- 172.21.15.163:0/3155790713 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09a001c5b0 con 0x7f09ac0fbac0 2024-03-11T22:54:58.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:58.904+0000 7f09b09ac700 1 -- 172.21.15.163:0/3155790713 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09ac0669f0 con 0x7f09ac0fbac0 2024-03-11T22:54:58.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:58.904+0000 7f09b09ac700 1 -- 172.21.15.163:0/3155790713 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09ac110370 con 0x7f09ac0fbac0 2024-03-11T22:54:58.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:58.905+0000 7f099affd700 1 -- 172.21.15.163:0/3155790713 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f09a0013a80 con 0x7f09ac0fbac0 2024-03-11T22:54:58.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:58.905+0000 7f09b09ac700 1 -- 172.21.15.163:0/3155790713 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09ac105890 con 0x7f09ac0fbac0 2024-03-11T22:54:58.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:58.905+0000 7f099affd700 1 --2- 172.21.15.163:0/3155790713 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f0994036820 0x7f0994038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:54:58.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:58.905+0000 7f099affd700 1 -- 172.21.15.163:0/3155790713 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f09a0049af0 con 0x7f09ac0fbac0 2024-03-11T22:54:58.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:58.906+0000 7f09a9d9b700 1 --2- 172.21.15.163:0/3155790713 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f0994036820 0x7f0994038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:54:58.911 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:58.906+0000 7f09a9d9b700 1 --2- 172.21.15.163:0/3155790713 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f0994036820 0x7f0994038cd0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f099c009990 tx=0x7f099c006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:54:58.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:58.910+0000 7f099affd700 1 -- 172.21.15.163:0/3155790713 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f09a0011960 con 0x7f09ac0fbac0 2024-03-11T22:54:59.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:59.169+0000 7f09b09ac700 1 -- 172.21.15.163:0/3155790713 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f09ac04bed0 con 0x7f09ac0fbac0 2024-03-11T22:54:59.173 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:59.169+0000 7f099affd700 1 -- 172.21.15.163:0/3155790713 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f09a0011370 con 0x7f09ac0fbac0 2024-03-11T22:54:59.174 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:54:59.174 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":1,"fsid":"fa913b96-dff9-11ee-95c8-87774f69a715","modified":"2024-03-11T22:53:11.271993Z","created":"2024-03-11T22:53:11.271993Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T22:54:59.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:59.171+0000 7f09b09ac700 1 -- 172.21.15.163:0/3155790713 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f0994036820 msgr2=0x7f0994038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:59.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:59.171+0000 7f09b09ac700 1 --2- 172.21.15.163:0/3155790713 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f0994036820 0x7f0994038cd0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f099c009990 tx=0x7f099c006e40 comp rx=0 tx=0).stop 2024-03-11T22:54:59.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:59.171+0000 7f09b09ac700 1 -- 172.21.15.163:0/3155790713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09ac0fbac0 msgr2=0x7f09ac10fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:54:59.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:59.171+0000 7f09b09ac700 1 --2- 172.21.15.163:0/3155790713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09ac0fbac0 0x7f09ac10fbf0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f09a0009680 tx=0x7f09a0009e30 comp rx=0 tx=0).stop 2024-03-11T22:54:59.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:59.171+0000 7f09b09ac700 1 -- 172.21.15.163:0/3155790713 shutdown_connections 2024-03-11T22:54:59.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:59.171+0000 7f09b09ac700 1 --2- 172.21.15.163:0/3155790713 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f0994036820 0x7f0994038cd0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:59.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:59.171+0000 7f09b09ac700 1 --2- 172.21.15.163:0/3155790713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09ac0fbac0 0x7f09ac10fbf0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:54:59.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:59.171+0000 7f09b09ac700 1 -- 172.21.15.163:0/3155790713 >> 172.21.15.163:0/3155790713 conn(0x7f09ac0f71a0 msgr2=0x7f09ac0f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:54:59.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:59.171+0000 7f09b09ac700 1 -- 172.21.15.163:0/3155790713 shutdown_connections 2024-03-11T22:54:59.176 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:54:59.171+0000 7f09b09ac700 1 -- 172.21.15.163:0/3155790713 wait complete. 2024-03-11T22:54:59.177 INFO:teuthology.orchestra.run.smithi163.stderr:dumped monmap epoch 1 2024-03-11T22:54:59.469 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:54:59 smithi153 ceph-mon[28962]: from='client.? 172.21.15.163:0/3155790713' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:55:00.318 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:00 smithi153 ceph-mon[28962]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:00.318 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:00 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:00.318 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:00 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:00.318 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:00 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:00.318 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:00 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:00.318 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:00 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:00.318 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:00 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:00.769 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-11T22:55:00.769 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:55:01.022 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-11T22:55:01.068 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-11T22:55:01.451 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:01 smithi153 ceph-mon[28962]: Deploying daemon prometheus.smithi153 on smithi153 2024-03-11T22:55:02.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.503+0000 7fea32953700 1 -- 172.21.15.163:0/4165989240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea2c0faea0 msgr2=0x7fea2c0fb2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:55:02.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.503+0000 7fea32953700 1 --2- 172.21.15.163:0/4165989240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea2c0faea0 0x7fea2c0fb2b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fea200048c0 tx=0x7fea20009e30 comp rx=0 tx=0).stop 2024-03-11T22:55:02.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.503+0000 7fea32953700 1 -- 172.21.15.163:0/4165989240 shutdown_connections 2024-03-11T22:55:02.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.503+0000 7fea32953700 1 --2- 172.21.15.163:0/4165989240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea2c0faea0 0x7fea2c0fb2b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:02.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.503+0000 7fea32953700 1 -- 172.21.15.163:0/4165989240 >> 172.21.15.163:0/4165989240 conn(0x7fea2c0f6920 msgr2=0x7fea2c0f8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:55:02.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.503+0000 7fea32953700 1 -- 172.21.15.163:0/4165989240 shutdown_connections 2024-03-11T22:55:02.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.503+0000 7fea32953700 1 -- 172.21.15.163:0/4165989240 wait complete. 2024-03-11T22:55:02.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.503+0000 7fea32953700 1 Processor -- start 2024-03-11T22:55:02.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.503+0000 7fea32953700 1 -- start start 2024-03-11T22:55:02.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.504+0000 7fea32953700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea2c0faea0 0x7fea2c10cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:55:02.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.504+0000 7fea32953700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea2c10d450 con 0x7fea2c0faea0 2024-03-11T22:55:02.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.504+0000 7fea2bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea2c0faea0 0x7fea2c10cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:55:02.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.504+0000 7fea2bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea2c0faea0 0x7fea2c10cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.163:43878/0 (socket says 172.21.15.163:43878) 2024-03-11T22:55:02.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.504+0000 7fea2bfff700 1 -- 172.21.15.163:0/4142110104 learned_addr learned my addr 172.21.15.163:0/4142110104 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-03-11T22:55:02.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.504+0000 7fea2bfff700 1 -- 172.21.15.163:0/4142110104 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea20005040 con 0x7fea2c0faea0 2024-03-11T22:55:02.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.504+0000 7fea2bfff700 1 --2- 172.21.15.163:0/4142110104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea2c0faea0 0x7fea2c10cf10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fea20009330 tx=0x7fea20009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:55:02.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.504+0000 7fea28ff9700 1 -- 172.21.15.163:0/4142110104 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea20004e60 con 0x7fea2c0faea0 2024-03-11T22:55:02.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.504+0000 7fea28ff9700 1 -- 172.21.15.163:0/4142110104 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fea200096b0 con 0x7fea2c0faea0 2024-03-11T22:55:02.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.504+0000 7fea32953700 1 -- 172.21.15.163:0/4142110104 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea2c10d650 con 0x7fea2c0faea0 2024-03-11T22:55:02.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.504+0000 7fea28ff9700 1 -- 172.21.15.163:0/4142110104 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea2001c5b0 con 0x7fea2c0faea0 2024-03-11T22:55:02.510 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.505+0000 7fea32953700 1 -- 172.21.15.163:0/4142110104 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea2c10e2f0 con 0x7fea2c0faea0 2024-03-11T22:55:02.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.505+0000 7fea28ff9700 1 -- 172.21.15.163:0/4142110104 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7fea20013420 con 0x7fea2c0faea0 2024-03-11T22:55:02.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.505+0000 7fea32953700 1 -- 172.21.15.163:0/4142110104 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea2c05f5c0 con 0x7fea2c0faea0 2024-03-11T22:55:02.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.505+0000 7fea28ff9700 1 --2- 172.21.15.163:0/4142110104 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7fea14036870 0x7fea14038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:55:02.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.506+0000 7fea28ff9700 1 -- 172.21.15.163:0/4142110104 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fea20049c50 con 0x7fea2c0faea0 2024-03-11T22:55:02.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.506+0000 7fea2b7fe700 1 --2- 172.21.15.163:0/4142110104 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7fea14036870 0x7fea14038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:55:02.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.506+0000 7fea2b7fe700 1 --2- 172.21.15.163:0/4142110104 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7fea14036870 0x7fea14038d20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fea1c009990 tx=0x7fea1c006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:55:02.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.510+0000 7fea28ff9700 1 -- 172.21.15.163:0/4142110104 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7fea20013980 con 0x7fea2c0faea0 2024-03-11T22:55:02.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:02 smithi153 ceph-mon[28962]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:02.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:02 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:02.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.766+0000 7fea32953700 1 -- 172.21.15.163:0/4142110104 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fea2c10ee40 con 0x7fea2c0faea0 2024-03-11T22:55:02.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.766+0000 7fea28ff9700 1 -- 172.21.15.163:0/4142110104 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fea20011370 con 0x7fea2c0faea0 2024-03-11T22:55:02.771 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:55:02.771 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":1,"fsid":"fa913b96-dff9-11ee-95c8-87774f69a715","modified":"2024-03-11T22:53:11.271993Z","created":"2024-03-11T22:53:11.271993Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T22:55:02.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.769+0000 7fea32953700 1 -- 172.21.15.163:0/4142110104 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7fea14036870 msgr2=0x7fea14038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:55:02.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.769+0000 7fea32953700 1 --2- 172.21.15.163:0/4142110104 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7fea14036870 0x7fea14038d20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fea1c009990 tx=0x7fea1c006e40 comp rx=0 tx=0).stop 2024-03-11T22:55:02.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.769+0000 7fea32953700 1 -- 172.21.15.163:0/4142110104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea2c0faea0 msgr2=0x7fea2c10cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:55:02.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.769+0000 7fea32953700 1 --2- 172.21.15.163:0/4142110104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea2c0faea0 0x7fea2c10cf10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fea20009330 tx=0x7fea20009e30 comp rx=0 tx=0).stop 2024-03-11T22:55:02.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.769+0000 7fea32953700 1 -- 172.21.15.163:0/4142110104 shutdown_connections 2024-03-11T22:55:02.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.769+0000 7fea32953700 1 --2- 172.21.15.163:0/4142110104 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7fea14036870 0x7fea14038d20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:02.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.769+0000 7fea32953700 1 --2- 172.21.15.163:0/4142110104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fea2c0faea0 0x7fea2c10cf10 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:02.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.769+0000 7fea32953700 1 -- 172.21.15.163:0/4142110104 >> 172.21.15.163:0/4142110104 conn(0x7fea2c0f6920 msgr2=0x7fea2c0f76d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:55:02.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.769+0000 7fea32953700 1 -- 172.21.15.163:0/4142110104 shutdown_connections 2024-03-11T22:55:02.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:02.769+0000 7fea32953700 1 -- 172.21.15.163:0/4142110104 wait complete. 2024-03-11T22:55:02.775 INFO:teuthology.orchestra.run.smithi163.stderr:dumped monmap epoch 1 2024-03-11T22:55:03.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:03 smithi153 ceph-mon[28962]: from='client.? 172.21.15.163:0/4142110104' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:55:04.354 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-11T22:55:04.354 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:55:04.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:04 smithi153 ceph-mon[28962]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:04.603 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-11T22:55:04.651 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-11T22:55:05.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:05.987+0000 7f2be6102700 1 -- 172.21.15.163:0/4104198794 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2be00fbad0 msgr2=0x7f2be00fbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:55:05.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:05.987+0000 7f2be6102700 1 --2- 172.21.15.163:0/4104198794 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2be00fbad0 0x7f2be00fbee0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2bd00048c0 tx=0x7f2bd0009e30 comp rx=0 tx=0).stop 2024-03-11T22:55:05.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:05.987+0000 7f2be6102700 1 -- 172.21.15.163:0/4104198794 shutdown_connections 2024-03-11T22:55:05.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:05.987+0000 7f2be6102700 1 --2- 172.21.15.163:0/4104198794 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2be00fbad0 0x7f2be00fbee0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:05.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:05.987+0000 7f2be6102700 1 -- 172.21.15.163:0/4104198794 >> 172.21.15.163:0/4104198794 conn(0x7f2be00f7190 msgr2=0x7f2be00f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:55:05.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:05.987+0000 7f2be6102700 1 -- 172.21.15.163:0/4104198794 shutdown_connections 2024-03-11T22:55:05.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:05.987+0000 7f2be6102700 1 -- 172.21.15.163:0/4104198794 wait complete. 2024-03-11T22:55:05.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:05.988+0000 7f2be6102700 1 Processor -- start 2024-03-11T22:55:05.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:05.988+0000 7f2be6102700 1 -- start start 2024-03-11T22:55:05.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:05.988+0000 7f2be6102700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2be00fbad0 0x7f2be0111fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:55:05.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:05.988+0000 7f2be6102700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2be010d9c0 con 0x7f2be00fbad0 2024-03-11T22:55:05.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:05.988+0000 7f2bdf7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2be00fbad0 0x7f2be0111fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:55:05.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:05.989+0000 7f2bdf7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2be00fbad0 0x7f2be0111fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.163:43900/0 (socket says 172.21.15.163:43900) 2024-03-11T22:55:05.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:05.989+0000 7f2bdf7fe700 1 -- 172.21.15.163:0/103066825 learned_addr learned my addr 172.21.15.163:0/103066825 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-03-11T22:55:05.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:05.989+0000 7f2bdf7fe700 1 -- 172.21.15.163:0/103066825 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bd0005040 con 0x7f2be00fbad0 2024-03-11T22:55:05.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:05.989+0000 7f2bdf7fe700 1 --2- 172.21.15.163:0/103066825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2be00fbad0 0x7f2be0111fd0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0009680 tx=0x7f2bd0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:55:05.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:05.989+0000 7f2bdcff9700 1 -- 172.21.15.163:0/103066825 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bd000bed0 con 0x7f2be00fbad0 2024-03-11T22:55:05.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:05.989+0000 7f2bdcff9700 1 -- 172.21.15.163:0/103066825 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2bd000b5c0 con 0x7f2be00fbad0 2024-03-11T22:55:05.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:05.989+0000 7f2be6102700 1 -- 172.21.15.163:0/103066825 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2be0065d60 con 0x7f2be00fbad0 2024-03-11T22:55:05.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:05.989+0000 7f2bdcff9700 1 -- 172.21.15.163:0/103066825 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2bd001c5b0 con 0x7f2be00fbad0 2024-03-11T22:55:05.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:05.989+0000 7f2be6102700 1 -- 172.21.15.163:0/103066825 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2be0112510 con 0x7f2be00fbad0 2024-03-11T22:55:05.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:05.990+0000 7f2bdcff9700 1 -- 172.21.15.163:0/103066825 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f2bd001c820 con 0x7f2be00fbad0 2024-03-11T22:55:05.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:05.990+0000 7f2bdcff9700 1 --2- 172.21.15.163:0/103066825 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f2bcc036820 0x7f2bcc038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:55:05.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:05.990+0000 7f2bdcff9700 1 -- 172.21.15.163:0/103066825 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f2bd0049b60 con 0x7f2be00fbad0 2024-03-11T22:55:05.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:05.990+0000 7f2be6102700 1 -- 172.21.15.163:0/103066825 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2be01058a0 con 0x7f2be00fbad0 2024-03-11T22:55:05.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:05.990+0000 7f2bd6ffd700 1 --2- 172.21.15.163:0/103066825 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f2bcc036820 0x7f2bcc038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:55:05.996 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:05.991+0000 7f2bd6ffd700 1 --2- 172.21.15.163:0/103066825 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f2bcc036820 0x7f2bcc038cd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2bc8009990 tx=0x7f2bc8006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:55:05.999 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:05.995+0000 7f2bdcff9700 1 -- 172.21.15.163:0/103066825 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f2bd0011dd0 con 0x7f2be00fbad0 2024-03-11T22:55:06.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:06.252+0000 7f2be6102700 1 -- 172.21.15.163:0/103066825 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2be002b340 con 0x7f2be00fbad0 2024-03-11T22:55:06.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:06.252+0000 7f2bdcff9700 1 -- 172.21.15.163:0/103066825 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f2bd0034090 con 0x7f2be00fbad0 2024-03-11T22:55:06.257 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:55:06.257 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":1,"fsid":"fa913b96-dff9-11ee-95c8-87774f69a715","modified":"2024-03-11T22:53:11.271993Z","created":"2024-03-11T22:53:11.271993Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T22:55:06.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:06.254+0000 7f2be6102700 1 -- 172.21.15.163:0/103066825 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f2bcc036820 msgr2=0x7f2bcc038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:55:06.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:06.254+0000 7f2be6102700 1 --2- 172.21.15.163:0/103066825 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f2bcc036820 0x7f2bcc038cd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2bc8009990 tx=0x7f2bc8006e40 comp rx=0 tx=0).stop 2024-03-11T22:55:06.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:06.254+0000 7f2be6102700 1 -- 172.21.15.163:0/103066825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2be00fbad0 msgr2=0x7f2be0111fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:55:06.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:06.254+0000 7f2be6102700 1 --2- 172.21.15.163:0/103066825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2be00fbad0 0x7f2be0111fd0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0009680 tx=0x7f2bd0009e30 comp rx=0 tx=0).stop 2024-03-11T22:55:06.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:06.254+0000 7f2be6102700 1 -- 172.21.15.163:0/103066825 shutdown_connections 2024-03-11T22:55:06.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:06.254+0000 7f2be6102700 1 --2- 172.21.15.163:0/103066825 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f2bcc036820 0x7f2bcc038cd0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:06.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:06.254+0000 7f2be6102700 1 --2- 172.21.15.163:0/103066825 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2be00fbad0 0x7f2be0111fd0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:06.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:06.254+0000 7f2be6102700 1 -- 172.21.15.163:0/103066825 >> 172.21.15.163:0/103066825 conn(0x7f2be00f7190 msgr2=0x7f2be00f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:55:06.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:06.254+0000 7f2be6102700 1 -- 172.21.15.163:0/103066825 shutdown_connections 2024-03-11T22:55:06.259 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:06.254+0000 7f2be6102700 1 -- 172.21.15.163:0/103066825 wait complete. 2024-03-11T22:55:06.260 INFO:teuthology.orchestra.run.smithi163.stderr:dumped monmap epoch 1 2024-03-11T22:55:06.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:06 smithi153 ceph-mon[28962]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:06.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:06 smithi153 ceph-mon[28962]: from='client.? 172.21.15.163:0/103066825' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:55:07.850 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-11T22:55:07.851 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:55:08.102 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-11T22:55:08.149 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-11T22:55:08.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:08 smithi153 ceph-mon[28962]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:09.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.643+0000 7f32492f2700 1 -- 172.21.15.163:0/1500271489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3244065db0 msgr2=0x7f32440ff7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:55:09.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.643+0000 7f32492f2700 1 --2- 172.21.15.163:0/1500271489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3244065db0 0x7f32440ff7c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3234009c60 tx=0x7f3234004d10 comp rx=0 tx=0).stop 2024-03-11T22:55:09.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.643+0000 7f32492f2700 1 -- 172.21.15.163:0/1500271489 shutdown_connections 2024-03-11T22:55:09.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.643+0000 7f32492f2700 1 --2- 172.21.15.163:0/1500271489 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3244065db0 0x7f32440ff7c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:09.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.643+0000 7f32492f2700 1 -- 172.21.15.163:0/1500271489 >> 172.21.15.163:0/1500271489 conn(0x7f32440f7190 msgr2=0x7f32440f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:55:09.649 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.643+0000 7f32492f2700 1 -- 172.21.15.163:0/1500271489 shutdown_connections 2024-03-11T22:55:09.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.643+0000 7f32492f2700 1 -- 172.21.15.163:0/1500271489 wait complete. 2024-03-11T22:55:09.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.644+0000 7f32492f2700 1 Processor -- start 2024-03-11T22:55:09.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.644+0000 7f32492f2700 1 -- start start 2024-03-11T22:55:09.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.644+0000 7f32492f2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3244065db0 0x7f324410ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:55:09.650 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.644+0000 7f32492f2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f324410d3d0 con 0x7f3244065db0 2024-03-11T22:55:09.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.644+0000 7f3242ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3244065db0 0x7f324410ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:55:09.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.644+0000 7f3242ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3244065db0 0x7f324410ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.163:51864/0 (socket says 172.21.15.163:51864) 2024-03-11T22:55:09.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.644+0000 7f3242ffd700 1 -- 172.21.15.163:0/2823203573 learned_addr learned my addr 172.21.15.163:0/2823203573 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-03-11T22:55:09.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.645+0000 7f3242ffd700 1 -- 172.21.15.163:0/2823203573 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3234009940 con 0x7f3244065db0 2024-03-11T22:55:09.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.645+0000 7f3242ffd700 1 --2- 172.21.15.163:0/2823203573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3244065db0 0x7f324410ce90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3234004e60 tx=0x7f3234004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:55:09.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.645+0000 7f323bfff700 1 -- 172.21.15.163:0/2823203573 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f323401d440 con 0x7f3244065db0 2024-03-11T22:55:09.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.645+0000 7f32492f2700 1 -- 172.21.15.163:0/2823203573 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f324410d5d0 con 0x7f3244065db0 2024-03-11T22:55:09.652 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.645+0000 7f323bfff700 1 -- 172.21.15.163:0/2823203573 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f323401da70 con 0x7f3244065db0 2024-03-11T22:55:09.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.645+0000 7f323bfff700 1 -- 172.21.15.163:0/2823203573 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3234012aa0 con 0x7f3244065db0 2024-03-11T22:55:09.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.645+0000 7f32492f2700 1 -- 172.21.15.163:0/2823203573 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f324410e270 con 0x7f3244065db0 2024-03-11T22:55:09.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.646+0000 7f32492f2700 1 -- 172.21.15.163:0/2823203573 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f324405f5c0 con 0x7f3244065db0 2024-03-11T22:55:09.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.646+0000 7f323bfff700 1 -- 172.21.15.163:0/2823203573 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f3234019040 con 0x7f3244065db0 2024-03-11T22:55:09.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.646+0000 7f323bfff700 1 --2- 172.21.15.163:0/2823203573 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f3230036870 0x7f3230038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:55:09.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.646+0000 7f323bfff700 1 -- 172.21.15.163:0/2823203573 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f323404a350 con 0x7f3244065db0 2024-03-11T22:55:09.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.646+0000 7f323a7fc700 1 --2- 172.21.15.163:0/2823203573 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f3230036870 0x7f3230038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:55:09.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.647+0000 7f323a7fc700 1 --2- 172.21.15.163:0/2823203573 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f3230036870 0x7f3230038d20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f322c009990 tx=0x7f322c006e40 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:55:09.659 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.654+0000 7f323bfff700 1 -- 172.21.15.163:0/2823203573 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f3234012410 con 0x7f3244065db0 2024-03-11T22:55:09.915 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.910+0000 7f32492f2700 1 -- 172.21.15.163:0/2823203573 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f324410ed60 con 0x7f3244065db0 2024-03-11T22:55:09.916 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.910+0000 7f323bfff700 1 -- 172.21.15.163:0/2823203573 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f3234017020 con 0x7f3244065db0 2024-03-11T22:55:09.916 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:55:09.916 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":1,"fsid":"fa913b96-dff9-11ee-95c8-87774f69a715","modified":"2024-03-11T22:53:11.271993Z","created":"2024-03-11T22:53:11.271993Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T22:55:09.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.912+0000 7f32492f2700 1 -- 172.21.15.163:0/2823203573 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f3230036870 msgr2=0x7f3230038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:55:09.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.912+0000 7f32492f2700 1 --2- 172.21.15.163:0/2823203573 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f3230036870 0x7f3230038d20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f322c009990 tx=0x7f322c006e40 comp rx=0 tx=0).stop 2024-03-11T22:55:09.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.912+0000 7f32492f2700 1 -- 172.21.15.163:0/2823203573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3244065db0 msgr2=0x7f324410ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:55:09.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.912+0000 7f32492f2700 1 --2- 172.21.15.163:0/2823203573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3244065db0 0x7f324410ce90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3234004e60 tx=0x7f3234004d10 comp rx=0 tx=0).stop 2024-03-11T22:55:09.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.912+0000 7f32492f2700 1 -- 172.21.15.163:0/2823203573 shutdown_connections 2024-03-11T22:55:09.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.912+0000 7f32492f2700 1 --2- 172.21.15.163:0/2823203573 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f3230036870 0x7f3230038d20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:09.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.912+0000 7f32492f2700 1 --2- 172.21.15.163:0/2823203573 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3244065db0 0x7f324410ce90 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:09.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.912+0000 7f32492f2700 1 -- 172.21.15.163:0/2823203573 >> 172.21.15.163:0/2823203573 conn(0x7f32440f7190 msgr2=0x7f32440f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:55:09.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.913+0000 7f32492f2700 1 -- 172.21.15.163:0/2823203573 shutdown_connections 2024-03-11T22:55:09.918 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:09.913+0000 7f32492f2700 1 -- 172.21.15.163:0/2823203573 wait complete. 2024-03-11T22:55:09.919 INFO:teuthology.orchestra.run.smithi163.stderr:dumped monmap epoch 1 2024-03-11T22:55:10.473 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:10 smithi153 ceph-mon[28962]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:10.474 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:10 smithi153 ceph-mon[28962]: from='client.? 172.21.15.163:0/2823203573' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:55:11.494 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-11T22:55:11.494 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:55:11.753 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-11T22:55:11.801 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-11T22:55:11.953 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:11 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:11.953 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:11 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:11.953 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:11 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:11.953 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:11 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T22:55:12.870 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:12 smithi153 ceph-mon[28962]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T22:55:12.870 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:12 smithi153 ceph-mon[28962]: mgrmap e14: smithi153.nyocep(active, since 54s) 2024-03-11T22:55:13.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.260+0000 7f85fe3a0700 1 -- 172.21.15.163:0/3858676982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85f80fd160 msgr2=0x7f85f80fd570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:55:13.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.260+0000 7f85fe3a0700 1 --2- 172.21.15.163:0/3858676982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85f80fd160 0x7f85f80fd570 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f85e80048c0 tx=0x7f85e8009e30 comp rx=0 tx=0).stop 2024-03-11T22:55:13.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.260+0000 7f85fe3a0700 1 -- 172.21.15.163:0/3858676982 shutdown_connections 2024-03-11T22:55:13.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.260+0000 7f85fe3a0700 1 --2- 172.21.15.163:0/3858676982 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85f80fd160 0x7f85f80fd570 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:13.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.260+0000 7f85fe3a0700 1 -- 172.21.15.163:0/3858676982 >> 172.21.15.163:0/3858676982 conn(0x7f85f80f6920 msgr2=0x7f85f80f8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:55:13.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.260+0000 7f85fe3a0700 1 -- 172.21.15.163:0/3858676982 shutdown_connections 2024-03-11T22:55:13.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.260+0000 7f85fe3a0700 1 -- 172.21.15.163:0/3858676982 wait complete. 2024-03-11T22:55:13.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.261+0000 7f85fe3a0700 1 Processor -- start 2024-03-11T22:55:13.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.261+0000 7f85fe3a0700 1 -- start start 2024-03-11T22:55:13.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.261+0000 7f85fe3a0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85f80fd160 0x7f85f810cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:55:13.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.261+0000 7f85fe3a0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f810d430 con 0x7f85f80fd160 2024-03-11T22:55:13.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.261+0000 7f85f7fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85f80fd160 0x7f85f810cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:55:13.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.261+0000 7f85f7fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85f80fd160 0x7f85f810cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.163:51882/0 (socket says 172.21.15.163:51882) 2024-03-11T22:55:13.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.261+0000 7f85f7fff700 1 -- 172.21.15.163:0/4072892998 learned_addr learned my addr 172.21.15.163:0/4072892998 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-03-11T22:55:13.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.261+0000 7f85f7fff700 1 -- 172.21.15.163:0/4072892998 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e8005040 con 0x7f85f80fd160 2024-03-11T22:55:13.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.262+0000 7f85f7fff700 1 --2- 172.21.15.163:0/4072892998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85f80fd160 0x7f85f810cef0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f85e8009330 tx=0x7f85e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:55:13.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.262+0000 7f85f4ff9700 1 -- 172.21.15.163:0/4072892998 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85e800be50 con 0x7f85f80fd160 2024-03-11T22:55:13.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.262+0000 7f85f4ff9700 1 -- 172.21.15.163:0/4072892998 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f85e80096b0 con 0x7f85f80fd160 2024-03-11T22:55:13.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.262+0000 7f85f4ff9700 1 -- 172.21.15.163:0/4072892998 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85e801c5b0 con 0x7f85f80fd160 2024-03-11T22:55:13.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.262+0000 7f85fe3a0700 1 -- 172.21.15.163:0/4072892998 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85f810d630 con 0x7f85f80fd160 2024-03-11T22:55:13.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.262+0000 7f85fe3a0700 1 -- 172.21.15.163:0/4072892998 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85f810e2d0 con 0x7f85f80fd160 2024-03-11T22:55:13.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.263+0000 7f85f4ff9700 1 -- 172.21.15.163:0/4072892998 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 14) v1 ==== 43439+0+0 (secure 0 0 0) 0x7f85e8013420 con 0x7f85f80fd160 2024-03-11T22:55:13.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.263+0000 7f85fe3a0700 1 -- 172.21.15.163:0/4072892998 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85f805f5c0 con 0x7f85f80fd160 2024-03-11T22:55:13.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.263+0000 7f85f4ff9700 1 --2- 172.21.15.163:0/4072892998 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f85e00368c0 0x7f85e0038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:55:13.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.263+0000 7f85f4ff9700 1 -- 172.21.15.163:0/4072892998 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f85e8049ae0 con 0x7f85f80fd160 2024-03-11T22:55:13.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.263+0000 7f85f77fe700 1 -- 172.21.15.163:0/4072892998 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f85e00368c0 msgr2=0x7f85e0038d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/4212724556 2024-03-11T22:55:13.270 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.263+0000 7f85f77fe700 1 --2- 172.21.15.163:0/4072892998 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f85e00368c0 0x7f85e0038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T22:55:13.273 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.268+0000 7f85f4ff9700 1 -- 172.21.15.163:0/4072892998 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f85e8011550 con 0x7f85f80fd160 2024-03-11T22:55:13.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.464+0000 7f85f77fe700 1 -- 172.21.15.163:0/4072892998 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f85e00368c0 msgr2=0x7f85e0038d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/4212724556 2024-03-11T22:55:13.469 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.464+0000 7f85f77fe700 1 --2- 172.21.15.163:0/4072892998 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f85e00368c0 0x7f85e0038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-11T22:55:13.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.525+0000 7f85fe3a0700 1 -- 172.21.15.163:0/4072892998 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f85f810ee20 con 0x7f85f80fd160 2024-03-11T22:55:13.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.526+0000 7f85f4ff9700 1 -- 172.21.15.163:0/4072892998 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f85e801ba30 con 0x7f85f80fd160 2024-03-11T22:55:13.531 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:55:13.531 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":1,"fsid":"fa913b96-dff9-11ee-95c8-87774f69a715","modified":"2024-03-11T22:53:11.271993Z","created":"2024-03-11T22:53:11.271993Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T22:55:13.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.528+0000 7f85fe3a0700 1 -- 172.21.15.163:0/4072892998 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f85e00368c0 msgr2=0x7f85e0038d70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T22:55:13.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.528+0000 7f85fe3a0700 1 --2- 172.21.15.163:0/4072892998 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f85e00368c0 0x7f85e0038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:13.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.528+0000 7f85fe3a0700 1 -- 172.21.15.163:0/4072892998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85f80fd160 msgr2=0x7f85f810cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:55:13.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.528+0000 7f85fe3a0700 1 --2- 172.21.15.163:0/4072892998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85f80fd160 0x7f85f810cef0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f85e8009330 tx=0x7f85e8009e30 comp rx=0 tx=0).stop 2024-03-11T22:55:13.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.528+0000 7f85fe3a0700 1 -- 172.21.15.163:0/4072892998 shutdown_connections 2024-03-11T22:55:13.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.528+0000 7f85fe3a0700 1 --2- 172.21.15.163:0/4072892998 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f85e00368c0 0x7f85e0038d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:13.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.528+0000 7f85fe3a0700 1 --2- 172.21.15.163:0/4072892998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85f80fd160 0x7f85f810cef0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:13.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.528+0000 7f85fe3a0700 1 -- 172.21.15.163:0/4072892998 >> 172.21.15.163:0/4072892998 conn(0x7f85f80f6920 msgr2=0x7f85f80fb520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:55:13.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.528+0000 7f85fe3a0700 1 -- 172.21.15.163:0/4072892998 shutdown_connections 2024-03-11T22:55:13.534 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:13.528+0000 7f85fe3a0700 1 -- 172.21.15.163:0/4072892998 wait complete. 2024-03-11T22:55:13.534 INFO:teuthology.orchestra.run.smithi163.stderr:dumped monmap epoch 1 2024-03-11T22:55:13.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:13 smithi153 ceph-mon[28962]: from='client.? 172.21.15.163:0/4072892998' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:55:15.031 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-11T22:55:15.031 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:55:15.281 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-11T22:55:15.329 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-11T22:55:16.678 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.672+0000 7f73a65dd700 1 -- 172.21.15.163:0/1801092612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73a00fbad0 msgr2=0x7f73a00fbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:55:16.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.672+0000 7f73a65dd700 1 --2- 172.21.15.163:0/1801092612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73a00fbad0 0x7f73a00fbee0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f73940048c0 tx=0x7f7394009e30 comp rx=0 tx=0).stop 2024-03-11T22:55:16.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.672+0000 7f73a65dd700 1 -- 172.21.15.163:0/1801092612 shutdown_connections 2024-03-11T22:55:16.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.672+0000 7f73a65dd700 1 --2- 172.21.15.163:0/1801092612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73a00fbad0 0x7f73a00fbee0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:16.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.672+0000 7f73a65dd700 1 -- 172.21.15.163:0/1801092612 >> 172.21.15.163:0/1801092612 conn(0x7f73a00f7190 msgr2=0x7f73a00f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:55:16.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.672+0000 7f73a65dd700 1 -- 172.21.15.163:0/1801092612 shutdown_connections 2024-03-11T22:55:16.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.672+0000 7f73a65dd700 1 -- 172.21.15.163:0/1801092612 wait complete. 2024-03-11T22:55:16.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.673+0000 7f73a65dd700 1 Processor -- start 2024-03-11T22:55:16.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.673+0000 7f73a65dd700 1 -- start start 2024-03-11T22:55:16.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.673+0000 7f73a65dd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73a00fbad0 0x7f73a00f9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:55:16.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.673+0000 7f73a65dd700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73a00f76d0 con 0x7f73a00fbad0 2024-03-11T22:55:16.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.673+0000 7f739ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73a00fbad0 0x7f73a00f9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:55:16.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.673+0000 7f739ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73a00fbad0 0x7f73a00f9250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.163:51890/0 (socket says 172.21.15.163:51890) 2024-03-11T22:55:16.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.673+0000 7f739ffff700 1 -- 172.21.15.163:0/3339061768 learned_addr learned my addr 172.21.15.163:0/3339061768 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-03-11T22:55:16.679 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.673+0000 7f739ffff700 1 -- 172.21.15.163:0/3339061768 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7394005040 con 0x7f73a00fbad0 2024-03-11T22:55:16.680 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.674+0000 7f739ffff700 1 --2- 172.21.15.163:0/3339061768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73a00fbad0 0x7f73a00f9250 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7394009b80 tx=0x7f7394009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:55:16.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.674+0000 7f739cff9700 1 -- 172.21.15.163:0/3339061768 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f739400b5c0 con 0x7f73a00fbad0 2024-03-11T22:55:16.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.674+0000 7f739cff9700 1 -- 172.21.15.163:0/3339061768 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f739400be70 con 0x7f73a00fbad0 2024-03-11T22:55:16.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.674+0000 7f73a65dd700 1 -- 172.21.15.163:0/3339061768 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73a00f78d0 con 0x7f73a00fbad0 2024-03-11T22:55:16.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.674+0000 7f739cff9700 1 -- 172.21.15.163:0/3339061768 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f739401c660 con 0x7f73a00fbad0 2024-03-11T22:55:16.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.674+0000 7f73a65dd700 1 -- 172.21.15.163:0/3339061768 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73a00f7dc0 con 0x7f73a00fbad0 2024-03-11T22:55:16.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.674+0000 7f739cff9700 1 -- 172.21.15.163:0/3339061768 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 14) v1 ==== 43439+0+0 (secure 0 0 0) 0x7f7394013420 con 0x7f73a00fbad0 2024-03-11T22:55:16.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.675+0000 7f73a65dd700 1 -- 172.21.15.163:0/3339061768 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73a01068b0 con 0x7f73a00fbad0 2024-03-11T22:55:16.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.675+0000 7f739cff9700 1 --2- 172.21.15.163:0/3339061768 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f73880368c0 0x7f7388038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:55:16.681 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.675+0000 7f739cff9700 1 -- 172.21.15.163:0/3339061768 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f7394049b40 con 0x7f73a00fbad0 2024-03-11T22:55:16.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.675+0000 7f739f7fe700 1 -- 172.21.15.163:0/3339061768 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f73880368c0 msgr2=0x7f7388038d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/4212724556 2024-03-11T22:55:16.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.675+0000 7f739f7fe700 1 --2- 172.21.15.163:0/3339061768 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f73880368c0 0x7f7388038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T22:55:16.685 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.680+0000 7f739cff9700 1 -- 172.21.15.163:0/3339061768 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f7394013b10 con 0x7f73a00fbad0 2024-03-11T22:55:16.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.876+0000 7f739f7fe700 1 -- 172.21.15.163:0/3339061768 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f73880368c0 msgr2=0x7f7388038d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/4212724556 2024-03-11T22:55:16.881 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.876+0000 7f739f7fe700 1 --2- 172.21.15.163:0/3339061768 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f73880368c0 0x7f7388038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-11T22:55:16.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.937+0000 7f73a65dd700 1 -- 172.21.15.163:0/3339061768 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f73a004bed0 con 0x7f73a00fbad0 2024-03-11T22:55:16.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.938+0000 7f739cff9700 1 -- 172.21.15.163:0/3339061768 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f7394011370 con 0x7f73a00fbad0 2024-03-11T22:55:16.943 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:55:16.943 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":1,"fsid":"fa913b96-dff9-11ee-95c8-87774f69a715","modified":"2024-03-11T22:53:11.271993Z","created":"2024-03-11T22:53:11.271993Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T22:55:16.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.940+0000 7f73a65dd700 1 -- 172.21.15.163:0/3339061768 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f73880368c0 msgr2=0x7f7388038d70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-11T22:55:16.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.940+0000 7f73a65dd700 1 --2- 172.21.15.163:0/3339061768 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f73880368c0 0x7f7388038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:16.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.940+0000 7f73a65dd700 1 -- 172.21.15.163:0/3339061768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73a00fbad0 msgr2=0x7f73a00f9250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:55:16.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.940+0000 7f73a65dd700 1 --2- 172.21.15.163:0/3339061768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73a00fbad0 0x7f73a00f9250 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7394009b80 tx=0x7f7394009e30 comp rx=0 tx=0).stop 2024-03-11T22:55:16.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.940+0000 7f73a65dd700 1 -- 172.21.15.163:0/3339061768 shutdown_connections 2024-03-11T22:55:16.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.940+0000 7f73a65dd700 1 --2- 172.21.15.163:0/3339061768 >> [v2:172.21.15.153:6800/4212724556,v1:172.21.15.153:6801/4212724556] conn(0x7f73880368c0 0x7f7388038d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:16.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.940+0000 7f73a65dd700 1 --2- 172.21.15.163:0/3339061768 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73a00fbad0 0x7f73a00f9250 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:16.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.940+0000 7f73a65dd700 1 -- 172.21.15.163:0/3339061768 >> 172.21.15.163:0/3339061768 conn(0x7f73a00f7190 msgr2=0x7f73a0065960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:55:16.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.940+0000 7f73a65dd700 1 -- 172.21.15.163:0/3339061768 shutdown_connections 2024-03-11T22:55:16.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:16.940+0000 7f73a65dd700 1 -- 172.21.15.163:0/3339061768 wait complete. 2024-03-11T22:55:16.946 INFO:teuthology.orchestra.run.smithi163.stderr:dumped monmap epoch 1 2024-03-11T22:55:17.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:16 smithi153 ceph-mon[28962]: from='client.? 172.21.15.163:0/3339061768' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:55:18.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:18 smithi153 ceph-mon[28962]: Active manager daemon smithi153.nyocep restarted 2024-03-11T22:55:18.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:18 smithi153 ceph-mon[28962]: Activating manager daemon smithi153.nyocep 2024-03-11T22:55:18.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:18 smithi153 ceph-mon[28962]: osdmap e5: 0 total, 0 up, 0 in 2024-03-11T22:55:18.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:18 smithi153 ceph-mon[28962]: mgrmap e15: smithi153.nyocep(active, starting, since 0.00297457s) 2024-03-11T22:55:18.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:18 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-03-11T22:55:18.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:18 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mgr metadata", "who": "smithi153.nyocep", "id": "smithi153.nyocep"}]: dispatch 2024-03-11T22:55:18.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:18 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:55:18.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:18 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:55:18.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:18 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:55:18.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:18 smithi153 ceph-mon[28962]: Manager daemon smithi153.nyocep is now available 2024-03-11T22:55:18.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:18 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:18.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:18 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:18.534 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-11T22:55:18.534 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:55:18.789 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-11T22:55:18.836 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-11T22:55:19.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:19 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:19.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:19 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:55:19.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:19 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.nyocep/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:55:19.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:19 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.nyocep/trash_purge_schedule"}]: dispatch 2024-03-11T22:55:19.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:19 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:19.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:19 smithi153 ceph-mon[28962]: mgrmap e16: smithi153.nyocep(active, since 1.00704s) 2024-03-11T22:55:20.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.320+0000 7f39440b7700 1 -- 172.21.15.163:0/3319938694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f393c0fcbd0 msgr2=0x7f393c0fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:55:20.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.320+0000 7f39440b7700 1 --2- 172.21.15.163:0/3319938694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f393c0fcbd0 0x7f393c0fcfe0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3934009c60 tx=0x7f3934004d10 comp rx=0 tx=0).stop 2024-03-11T22:55:20.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.320+0000 7f39440b7700 1 -- 172.21.15.163:0/3319938694 shutdown_connections 2024-03-11T22:55:20.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.320+0000 7f39440b7700 1 --2- 172.21.15.163:0/3319938694 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f393c0fcbd0 0x7f393c0fcfe0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:20.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.320+0000 7f39440b7700 1 -- 172.21.15.163:0/3319938694 >> 172.21.15.163:0/3319938694 conn(0x7f393c0f8270 msgr2=0x7f393c0fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:55:20.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.320+0000 7f39440b7700 1 -- 172.21.15.163:0/3319938694 shutdown_connections 2024-03-11T22:55:20.330 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.320+0000 7f39440b7700 1 -- 172.21.15.163:0/3319938694 wait complete. 2024-03-11T22:55:20.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.321+0000 7f39440b7700 1 Processor -- start 2024-03-11T22:55:20.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.321+0000 7f39440b7700 1 -- start start 2024-03-11T22:55:20.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.321+0000 7f39440b7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f393c0fcbd0 0x7f393c10cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:55:20.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.321+0000 7f39440b7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393c10d4a0 con 0x7f393c0fcbd0 2024-03-11T22:55:20.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.321+0000 7f3941e53700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f393c0fcbd0 0x7f393c10cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:55:20.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.321+0000 7f3941e53700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f393c0fcbd0 0x7f393c10cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.163:58908/0 (socket says 172.21.15.163:58908) 2024-03-11T22:55:20.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.321+0000 7f3941e53700 1 -- 172.21.15.163:0/3179057404 learned_addr learned my addr 172.21.15.163:0/3179057404 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-03-11T22:55:20.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.321+0000 7f3941e53700 1 -- 172.21.15.163:0/3179057404 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3934009940 con 0x7f393c0fcbd0 2024-03-11T22:55:20.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.322+0000 7f3941e53700 1 --2- 172.21.15.163:0/3179057404 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f393c0fcbd0 0x7f393c10cf60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3934015040 tx=0x7f3934004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:55:20.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.322+0000 7f393a7fc700 1 -- 172.21.15.163:0/3179057404 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3934007c80 con 0x7f393c0fcbd0 2024-03-11T22:55:20.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.322+0000 7f393a7fc700 1 -- 172.21.15.163:0/3179057404 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f393401d910 con 0x7f393c0fcbd0 2024-03-11T22:55:20.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.322+0000 7f393a7fc700 1 -- 172.21.15.163:0/3179057404 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39340129f0 con 0x7f393c0fcbd0 2024-03-11T22:55:20.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.322+0000 7f39440b7700 1 -- 172.21.15.163:0/3179057404 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f393c10d6a0 con 0x7f393c0fcbd0 2024-03-11T22:55:20.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.322+0000 7f39440b7700 1 -- 172.21.15.163:0/3179057404 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f393c10e2f0 con 0x7f393c0fcbd0 2024-03-11T22:55:20.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.323+0000 7f393a7fc700 1 -- 172.21.15.163:0/3179057404 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 16) v1 ==== 43289+0+0 (secure 0 0 0) 0x7f3934019040 con 0x7f393c0fcbd0 2024-03-11T22:55:20.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.323+0000 7f39440b7700 1 -- 172.21.15.163:0/3179057404 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f393c05f5c0 con 0x7f393c0fcbd0 2024-03-11T22:55:20.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.323+0000 7f393a7fc700 1 --2- 172.21.15.163:0/3179057404 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f3924036750 0x7f3924038c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:55:20.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.323+0000 7f393a7fc700 1 -- 172.21.15.163:0/3179057404 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f393404a430 con 0x7f393c0fcbd0 2024-03-11T22:55:20.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.323+0000 7f3941652700 1 --2- 172.21.15.163:0/3179057404 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f3924036750 0x7f3924038c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:55:20.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.323+0000 7f3941652700 1 --2- 172.21.15.163:0/3179057404 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f3924036750 0x7f3924038c00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3930009990 tx=0x7f3930006e40 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:55:20.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.328+0000 7f393a7fc700 1 -- 172.21.15.163:0/3179057404 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f393402ea50 con 0x7f393c0fcbd0 2024-03-11T22:55:20.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:20 smithi153 ceph-mon[28962]: [11/Mar/2024:22:55:19] ENGINE Bus STARTING 2024-03-11T22:55:20.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:20 smithi153 ceph-mon[28962]: [11/Mar/2024:22:55:19] ENGINE Serving on https://172.21.15.153:7150 2024-03-11T22:55:20.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:20 smithi153 ceph-mon[28962]: [11/Mar/2024:22:55:19] ENGINE Bus STARTED 2024-03-11T22:55:20.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:20 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:20.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:20 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:20.604 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.599+0000 7f39440b7700 1 -- 172.21.15.163:0/3179057404 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f393c10ede0 con 0x7f393c0fcbd0 2024-03-11T22:55:20.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.600+0000 7f393a7fc700 1 -- 172.21.15.163:0/3179057404 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f3934017030 con 0x7f393c0fcbd0 2024-03-11T22:55:20.605 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:55:20.605 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":1,"fsid":"fa913b96-dff9-11ee-95c8-87774f69a715","modified":"2024-03-11T22:53:11.271993Z","created":"2024-03-11T22:53:11.271993Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T22:55:20.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.602+0000 7f39440b7700 1 -- 172.21.15.163:0/3179057404 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f3924036750 msgr2=0x7f3924038c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:55:20.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.602+0000 7f39440b7700 1 --2- 172.21.15.163:0/3179057404 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f3924036750 0x7f3924038c00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3930009990 tx=0x7f3930006e40 comp rx=0 tx=0).stop 2024-03-11T22:55:20.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.602+0000 7f39440b7700 1 -- 172.21.15.163:0/3179057404 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f393c0fcbd0 msgr2=0x7f393c10cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:55:20.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.602+0000 7f39440b7700 1 --2- 172.21.15.163:0/3179057404 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f393c0fcbd0 0x7f393c10cf60 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3934015040 tx=0x7f3934004d10 comp rx=0 tx=0).stop 2024-03-11T22:55:20.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.602+0000 7f39440b7700 1 -- 172.21.15.163:0/3179057404 shutdown_connections 2024-03-11T22:55:20.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.602+0000 7f39440b7700 1 --2- 172.21.15.163:0/3179057404 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f3924036750 0x7f3924038c00 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:20.607 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.602+0000 7f39440b7700 1 --2- 172.21.15.163:0/3179057404 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f393c0fcbd0 0x7f393c10cf60 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:20.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.602+0000 7f39440b7700 1 -- 172.21.15.163:0/3179057404 >> 172.21.15.163:0/3179057404 conn(0x7f393c0f8270 msgr2=0x7f393c0f9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:55:20.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.602+0000 7f39440b7700 1 -- 172.21.15.163:0/3179057404 shutdown_connections 2024-03-11T22:55:20.608 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:20.602+0000 7f39440b7700 1 -- 172.21.15.163:0/3179057404 wait complete. 2024-03-11T22:55:20.608 INFO:teuthology.orchestra.run.smithi163.stderr:dumped monmap epoch 1 2024-03-11T22:55:21.904 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:21 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:21.904 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:21 smithi153 ceph-mon[28962]: from='client.? 172.21.15.163:0/3179057404' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:55:21.905 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:21 smithi153 ceph-mon[28962]: mgrmap e17: smithi153.nyocep(active, since 2s) 2024-03-11T22:55:22.202 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-11T22:55:22.203 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:55:22.789 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-11T22:55:22.844 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-11T22:55:23.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:23 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:23.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:23 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:23.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:23 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:23.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:23 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:23.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:23 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config rm", "who": "osd/host:smithi153", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:55:25.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.113+0000 7f4bb06e2700 1 -- 172.21.15.163:0/2568721663 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ba8102bd0 msgr2=0x7f4ba8102fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:55:25.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.113+0000 7f4bb06e2700 1 --2- 172.21.15.163:0/2568721663 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ba8102bd0 0x7f4ba8102fa0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4b980048c0 tx=0x7f4b98009e30 comp rx=0 tx=0).stop 2024-03-11T22:55:25.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.113+0000 7f4bb06e2700 1 -- 172.21.15.163:0/2568721663 shutdown_connections 2024-03-11T22:55:25.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.113+0000 7f4bb06e2700 1 --2- 172.21.15.163:0/2568721663 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ba8102bd0 0x7f4ba8102fa0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:25.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.113+0000 7f4bb06e2700 1 -- 172.21.15.163:0/2568721663 >> 172.21.15.163:0/2568721663 conn(0x7f4ba80f8760 msgr2=0x7f4ba80fab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:55:25.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.113+0000 7f4bb06e2700 1 -- 172.21.15.163:0/2568721663 shutdown_connections 2024-03-11T22:55:25.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.113+0000 7f4bb06e2700 1 -- 172.21.15.163:0/2568721663 wait complete. 2024-03-11T22:55:25.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.114+0000 7f4bb06e2700 1 Processor -- start 2024-03-11T22:55:25.120 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.114+0000 7f4bb06e2700 1 -- start start 2024-03-11T22:55:25.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.114+0000 7f4bb06e2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ba8102bd0 0x7f4ba8112f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:55:25.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.114+0000 7f4bb06e2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba8113460 con 0x7f4ba8102bd0 2024-03-11T22:55:25.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.114+0000 7f4bae47e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ba8102bd0 0x7f4ba8112f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:55:25.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.114+0000 7f4bae47e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ba8102bd0 0x7f4ba8112f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.163:58920/0 (socket says 172.21.15.163:58920) 2024-03-11T22:55:25.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.114+0000 7f4bae47e700 1 -- 172.21.15.163:0/2672378778 learned_addr learned my addr 172.21.15.163:0/2672378778 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-03-11T22:55:25.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.114+0000 7f4bae47e700 1 -- 172.21.15.163:0/2672378778 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b98005040 con 0x7f4ba8102bd0 2024-03-11T22:55:25.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.114+0000 7f4bae47e700 1 --2- 172.21.15.163:0/2672378778 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ba8102bd0 0x7f4ba8112f20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4b98009330 tx=0x7f4b98009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:55:25.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.115+0000 7f4b9effd700 1 -- 172.21.15.163:0/2672378778 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b9800be50 con 0x7f4ba8102bd0 2024-03-11T22:55:25.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.115+0000 7f4b9effd700 1 -- 172.21.15.163:0/2672378778 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4b98009630 con 0x7f4ba8102bd0 2024-03-11T22:55:25.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.115+0000 7f4bb06e2700 1 -- 172.21.15.163:0/2672378778 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ba8113660 con 0x7f4ba8102bd0 2024-03-11T22:55:25.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.115+0000 7f4bb06e2700 1 -- 172.21.15.163:0/2672378778 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba8113b50 con 0x7f4ba8102bd0 2024-03-11T22:55:25.121 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.115+0000 7f4b9effd700 1 -- 172.21.15.163:0/2672378778 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b9801c3f0 con 0x7f4ba8102bd0 2024-03-11T22:55:25.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.116+0000 7f4b9effd700 1 -- 172.21.15.163:0/2672378778 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) v1 ==== 43483+0+0 (secure 0 0 0) 0x7f4b9801c6f0 con 0x7f4ba8102bd0 2024-03-11T22:55:25.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.116+0000 7f4bb06e2700 1 -- 172.21.15.163:0/2672378778 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ba805f5c0 con 0x7f4ba8102bd0 2024-03-11T22:55:25.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.116+0000 7f4b9effd700 1 --2- 172.21.15.163:0/2672378778 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f4b94036960 0x7f4b94038e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:55:25.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.116+0000 7f4b9effd700 1 -- 172.21.15.163:0/2672378778 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f4b98049b70 con 0x7f4ba8102bd0 2024-03-11T22:55:25.122 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.116+0000 7f4badc7d700 1 --2- 172.21.15.163:0/2672378778 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f4b94036960 0x7f4b94038e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:55:25.123 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.116+0000 7f4badc7d700 1 --2- 172.21.15.163:0/2672378778 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f4b94036960 0x7f4b94038e10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4009990 tx=0x7f4ba4006e40 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:55:25.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.121+0000 7f4b9effd700 1 -- 172.21.15.163:0/2672378778 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f4b98025400 con 0x7f4ba8102bd0 2024-03-11T22:55:25.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.400+0000 7f4bb06e2700 1 -- 172.21.15.163:0/2672378778 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4ba810e6d0 con 0x7f4ba8102bd0 2024-03-11T22:55:25.406 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.401+0000 7f4b9effd700 1 -- 172.21.15.163:0/2672378778 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f4b98049380 con 0x7f4ba8102bd0 2024-03-11T22:55:25.407 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:55:25.407 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":1,"fsid":"fa913b96-dff9-11ee-95c8-87774f69a715","modified":"2024-03-11T22:53:11.271993Z","created":"2024-03-11T22:53:11.271993Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T22:55:25.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.403+0000 7f4bb06e2700 1 -- 172.21.15.163:0/2672378778 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f4b94036960 msgr2=0x7f4b94038e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:55:25.409 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.403+0000 7f4bb06e2700 1 --2- 172.21.15.163:0/2672378778 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f4b94036960 0x7f4b94038e10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4009990 tx=0x7f4ba4006e40 comp rx=0 tx=0).stop 2024-03-11T22:55:25.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.403+0000 7f4bb06e2700 1 -- 172.21.15.163:0/2672378778 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ba8102bd0 msgr2=0x7f4ba8112f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:55:25.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.403+0000 7f4bb06e2700 1 --2- 172.21.15.163:0/2672378778 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ba8102bd0 0x7f4ba8112f20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f4b98009330 tx=0x7f4b98009e30 comp rx=0 tx=0).stop 2024-03-11T22:55:25.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.403+0000 7f4bb06e2700 1 -- 172.21.15.163:0/2672378778 shutdown_connections 2024-03-11T22:55:25.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.403+0000 7f4bb06e2700 1 --2- 172.21.15.163:0/2672378778 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f4b94036960 0x7f4b94038e10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:25.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.403+0000 7f4bb06e2700 1 --2- 172.21.15.163:0/2672378778 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4ba8102bd0 0x7f4ba8112f20 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:25.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.403+0000 7f4bb06e2700 1 -- 172.21.15.163:0/2672378778 >> 172.21.15.163:0/2672378778 conn(0x7f4ba80f8760 msgr2=0x7f4ba80fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:55:25.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.403+0000 7f4bb06e2700 1 -- 172.21.15.163:0/2672378778 shutdown_connections 2024-03-11T22:55:25.410 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:25.403+0000 7f4bb06e2700 1 -- 172.21.15.163:0/2672378778 wait complete. 2024-03-11T22:55:25.410 INFO:teuthology.orchestra.run.smithi163.stderr:dumped monmap epoch 1 2024-03-11T22:55:25.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:25 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:25.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:25 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:25.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:25 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:25.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:25 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:25.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:25 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config rm", "who": "osd/host:smithi163", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:55:25.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:25 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:25.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:25 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:25.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:25 smithi153 ceph-mon[28962]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-11T22:55:25.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:25 smithi153 ceph-mon[28962]: Updating smithi163:/etc/ceph/ceph.conf 2024-03-11T22:55:25.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:25 smithi153 ceph-mon[28962]: Updating smithi163:/var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:55:25.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:25 smithi153 ceph-mon[28962]: Updating smithi153:/var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:55:25.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:25 smithi153 ceph-mon[28962]: Updating smithi163:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:55:25.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:25 smithi153 ceph-mon[28962]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:55:25.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:25 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:25.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:25 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:25.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:25 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:25.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:25 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:25.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:25 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:25.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:25 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi163", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-11T22:55:25.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:25 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi163", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-11T22:55:25.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:25 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:26.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:26 smithi153 ceph-mon[28962]: Updating smithi163:/var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:55:26.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:26 smithi153 ceph-mon[28962]: Updating smithi153:/var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:55:26.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:26 smithi153 ceph-mon[28962]: Deploying daemon crash.smithi163 on smithi163 2024-03-11T22:55:26.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:26 smithi153 ceph-mon[28962]: from='client.? 172.21.15.163:0/2672378778' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:55:27.535 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-11T22:55:27.535 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:55:28.290 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:55:29.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:29 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:29.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:29 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:29.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:29 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:29.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:29 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:29.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:29 smithi153 ceph-mon[28962]: Deploying daemon node-exporter.smithi163 on smithi163 2024-03-11T22:55:30.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.027+0000 7fda659aa700 1 -- 172.21.15.163:0/2482050189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda60101b60 msgr2=0x7fda60101f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:55:30.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.027+0000 7fda659aa700 1 --2- 172.21.15.163:0/2482050189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda60101b60 0x7fda60101f30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fda500048c0 tx=0x7fda50009e30 comp rx=0 tx=0).stop 2024-03-11T22:55:30.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.027+0000 7fda659aa700 1 -- 172.21.15.163:0/2482050189 shutdown_connections 2024-03-11T22:55:30.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.027+0000 7fda659aa700 1 --2- 172.21.15.163:0/2482050189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda60101b60 0x7fda60101f30 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:30.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.027+0000 7fda659aa700 1 -- 172.21.15.163:0/2482050189 >> 172.21.15.163:0/2482050189 conn(0x7fda60072850 msgr2=0x7fda60072c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:55:30.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.027+0000 7fda659aa700 1 -- 172.21.15.163:0/2482050189 shutdown_connections 2024-03-11T22:55:30.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.027+0000 7fda659aa700 1 -- 172.21.15.163:0/2482050189 wait complete. 2024-03-11T22:55:30.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.028+0000 7fda659aa700 1 Processor -- start 2024-03-11T22:55:30.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.028+0000 7fda659aa700 1 -- start start 2024-03-11T22:55:30.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.028+0000 7fda659aa700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda60101b60 0x7fda60115110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:55:30.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.028+0000 7fda659aa700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda60115650 con 0x7fda60101b60 2024-03-11T22:55:30.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.028+0000 7fda5effd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda60101b60 0x7fda60115110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:55:30.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.028+0000 7fda5effd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda60101b60 0x7fda60115110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.163:46626/0 (socket says 172.21.15.163:46626) 2024-03-11T22:55:30.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.028+0000 7fda5effd700 1 -- 172.21.15.163:0/3214293718 learned_addr learned my addr 172.21.15.163:0/3214293718 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-03-11T22:55:30.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.029+0000 7fda5effd700 1 -- 172.21.15.163:0/3214293718 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda50005040 con 0x7fda60101b60 2024-03-11T22:55:30.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.029+0000 7fda5effd700 1 --2- 172.21.15.163:0/3214293718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda60101b60 0x7fda60115110 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fda50009300 tx=0x7fda50009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:55:30.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.029+0000 7fda57fff700 1 -- 172.21.15.163:0/3214293718 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda50004e60 con 0x7fda60101b60 2024-03-11T22:55:30.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.029+0000 7fda57fff700 1 -- 172.21.15.163:0/3214293718 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fda50013420 con 0x7fda60101b60 2024-03-11T22:55:30.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.029+0000 7fda659aa700 1 -- 172.21.15.163:0/3214293718 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda60115850 con 0x7fda60101b60 2024-03-11T22:55:30.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.029+0000 7fda57fff700 1 -- 172.21.15.163:0/3214293718 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda5001c590 con 0x7fda60101b60 2024-03-11T22:55:30.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.029+0000 7fda659aa700 1 -- 172.21.15.163:0/3214293718 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda60115d40 con 0x7fda60101b60 2024-03-11T22:55:30.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.030+0000 7fda57fff700 1 -- 172.21.15.163:0/3214293718 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) v1 ==== 43483+0+0 (secure 0 0 0) 0x7fda50009600 con 0x7fda60101b60 2024-03-11T22:55:30.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.030+0000 7fda659aa700 1 -- 172.21.15.163:0/3214293718 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda6005f5c0 con 0x7fda60101b60 2024-03-11T22:55:30.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.030+0000 7fda57fff700 1 --2- 172.21.15.163:0/3214293718 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fda4c036910 0x7fda4c038dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:55:30.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.030+0000 7fda57fff700 1 -- 172.21.15.163:0/3214293718 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fda50049d10 con 0x7fda60101b60 2024-03-11T22:55:30.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.030+0000 7fda567fc700 1 --2- 172.21.15.163:0/3214293718 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fda4c036910 0x7fda4c038dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:55:30.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.031+0000 7fda567fc700 1 --2- 172.21.15.163:0/3214293718 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fda4c036910 0x7fda4c038dc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fda48009990 tx=0x7fda48006e40 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:55:30.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.035+0000 7fda57fff700 1 -- 172.21.15.163:0/3214293718 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7fda500119c0 con 0x7fda60101b60 2024-03-11T22:55:30.298 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.294+0000 7fda659aa700 1 -- 172.21.15.163:0/3214293718 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fda60110880 con 0x7fda60101b60 2024-03-11T22:55:30.299 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.294+0000 7fda57fff700 1 -- 172.21.15.163:0/3214293718 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fda50011370 con 0x7fda60101b60 2024-03-11T22:55:30.299 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:55:30.299 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":1,"fsid":"fa913b96-dff9-11ee-95c8-87774f69a715","modified":"2024-03-11T22:53:11.271993Z","created":"2024-03-11T22:53:11.271993Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T22:55:30.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.297+0000 7fda659aa700 1 -- 172.21.15.163:0/3214293718 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fda4c036910 msgr2=0x7fda4c038dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:55:30.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.297+0000 7fda659aa700 1 --2- 172.21.15.163:0/3214293718 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fda4c036910 0x7fda4c038dc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fda48009990 tx=0x7fda48006e40 comp rx=0 tx=0).stop 2024-03-11T22:55:30.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.297+0000 7fda659aa700 1 -- 172.21.15.163:0/3214293718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda60101b60 msgr2=0x7fda60115110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:55:30.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.297+0000 7fda659aa700 1 --2- 172.21.15.163:0/3214293718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda60101b60 0x7fda60115110 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fda50009300 tx=0x7fda50009e30 comp rx=0 tx=0).stop 2024-03-11T22:55:30.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.297+0000 7fda659aa700 1 -- 172.21.15.163:0/3214293718 shutdown_connections 2024-03-11T22:55:30.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.297+0000 7fda659aa700 1 --2- 172.21.15.163:0/3214293718 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fda4c036910 0x7fda4c038dc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:30.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.297+0000 7fda659aa700 1 --2- 172.21.15.163:0/3214293718 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda60101b60 0x7fda60115110 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:30.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.297+0000 7fda659aa700 1 -- 172.21.15.163:0/3214293718 >> 172.21.15.163:0/3214293718 conn(0x7fda60072850 msgr2=0x7fda60103bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:55:30.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.297+0000 7fda659aa700 1 -- 172.21.15.163:0/3214293718 shutdown_connections 2024-03-11T22:55:30.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:30.298+0000 7fda659aa700 1 -- 172.21.15.163:0/3214293718 wait complete. 2024-03-11T22:55:30.303 INFO:teuthology.orchestra.run.smithi163.stderr:dumped monmap epoch 1 2024-03-11T22:55:30.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:30 smithi153 ceph-mon[28962]: from='client.? 172.21.15.163:0/3214293718' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:55:31.860 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-11T22:55:31.860 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:55:32.121 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:55:34.385 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:34 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:34.385 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:34 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:55:34.385 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:34 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:34.385 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:34 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:34.529 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.524+0000 7f09e6ec0700 1 -- 172.21.15.163:0/3827033859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09e0101b60 msgr2=0x7f09e0101f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:55:34.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.524+0000 7f09e6ec0700 1 --2- 172.21.15.163:0/3827033859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09e0101b60 0x7f09e0101f30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f09d40048c0 tx=0x7f09d4009e30 comp rx=0 tx=0).stop 2024-03-11T22:55:34.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.524+0000 7f09e6ec0700 1 -- 172.21.15.163:0/3827033859 shutdown_connections 2024-03-11T22:55:34.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.524+0000 7f09e6ec0700 1 --2- 172.21.15.163:0/3827033859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09e0101b60 0x7f09e0101f30 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:34.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.524+0000 7f09e6ec0700 1 -- 172.21.15.163:0/3827033859 >> 172.21.15.163:0/3827033859 conn(0x7f09e0072850 msgr2=0x7f09e0072c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:55:34.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.524+0000 7f09e6ec0700 1 -- 172.21.15.163:0/3827033859 shutdown_connections 2024-03-11T22:55:34.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.524+0000 7f09e6ec0700 1 -- 172.21.15.163:0/3827033859 wait complete. 2024-03-11T22:55:34.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.525+0000 7f09e6ec0700 1 Processor -- start 2024-03-11T22:55:34.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.525+0000 7f09e6ec0700 1 -- start start 2024-03-11T22:55:34.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.525+0000 7f09e6ec0700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09e0101b60 0x7f09e0112eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:55:34.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.525+0000 7f09e6ec0700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e01133f0 con 0x7f09e0101b60 2024-03-11T22:55:34.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.525+0000 7f09e4c5c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09e0101b60 0x7f09e0112eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:55:34.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.525+0000 7f09e4c5c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09e0101b60 0x7f09e0112eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.163:46650/0 (socket says 172.21.15.163:46650) 2024-03-11T22:55:34.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.525+0000 7f09e4c5c700 1 -- 172.21.15.163:0/1223605517 learned_addr learned my addr 172.21.15.163:0/1223605517 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-03-11T22:55:34.531 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.526+0000 7f09e4c5c700 1 -- 172.21.15.163:0/1223605517 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d4005040 con 0x7f09e0101b60 2024-03-11T22:55:34.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.526+0000 7f09e4c5c700 1 --2- 172.21.15.163:0/1223605517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09e0101b60 0x7f09e0112eb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f09d4009300 tx=0x7f09d4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:55:34.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.526+0000 7f09dd7fa700 1 -- 172.21.15.163:0/1223605517 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09d4004e60 con 0x7f09e0101b60 2024-03-11T22:55:34.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.526+0000 7f09dd7fa700 1 -- 172.21.15.163:0/1223605517 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f09d4013420 con 0x7f09e0101b60 2024-03-11T22:55:34.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.526+0000 7f09e6ec0700 1 -- 172.21.15.163:0/1223605517 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09e01135f0 con 0x7f09e0101b60 2024-03-11T22:55:34.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.526+0000 7f09dd7fa700 1 -- 172.21.15.163:0/1223605517 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09d401c590 con 0x7f09e0101b60 2024-03-11T22:55:34.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.526+0000 7f09e6ec0700 1 -- 172.21.15.163:0/1223605517 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09e0113ae0 con 0x7f09e0101b60 2024-03-11T22:55:34.532 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.527+0000 7f09dd7fa700 1 -- 172.21.15.163:0/1223605517 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) v1 ==== 43483+0+0 (secure 0 0 0) 0x7f09d401c850 con 0x7f09e0101b60 2024-03-11T22:55:34.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.527+0000 7f09e6ec0700 1 -- 172.21.15.163:0/1223605517 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09e005f5c0 con 0x7f09e0101b60 2024-03-11T22:55:34.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.527+0000 7f09dd7fa700 1 --2- 172.21.15.163:0/1223605517 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f09d0036910 0x7f09d0038dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:55:34.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.527+0000 7f09dd7fa700 1 -- 172.21.15.163:0/1223605517 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f09d4049de0 con 0x7f09e0101b60 2024-03-11T22:55:34.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.527+0000 7f09dffff700 1 --2- 172.21.15.163:0/1223605517 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f09d0036910 0x7f09d0038dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:55:34.533 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.528+0000 7f09dffff700 1 --2- 172.21.15.163:0/1223605517 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f09d0036910 0x7f09d0038dc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f09cc009990 tx=0x7f09cc006e40 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:55:34.536 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.532+0000 7f09dd7fa700 1 -- 172.21.15.163:0/1223605517 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f09d4023300 con 0x7f09e0101b60 2024-03-11T22:55:34.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.833+0000 7f09e6ec0700 1 -- 172.21.15.163:0/1223605517 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f09e010e660 con 0x7f09e0101b60 2024-03-11T22:55:34.838 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.833+0000 7f09dd7fa700 1 -- 172.21.15.163:0/1223605517 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f09d4011b60 con 0x7f09e0101b60 2024-03-11T22:55:34.838 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:55:34.838 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":1,"fsid":"fa913b96-dff9-11ee-95c8-87774f69a715","modified":"2024-03-11T22:53:11.271993Z","created":"2024-03-11T22:53:11.271993Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T22:55:34.840 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.836+0000 7f09e6ec0700 1 -- 172.21.15.163:0/1223605517 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f09d0036910 msgr2=0x7f09d0038dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:55:34.840 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.836+0000 7f09e6ec0700 1 --2- 172.21.15.163:0/1223605517 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f09d0036910 0x7f09d0038dc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f09cc009990 tx=0x7f09cc006e40 comp rx=0 tx=0).stop 2024-03-11T22:55:34.840 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.836+0000 7f09e6ec0700 1 -- 172.21.15.163:0/1223605517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09e0101b60 msgr2=0x7f09e0112eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:55:34.840 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.836+0000 7f09e6ec0700 1 --2- 172.21.15.163:0/1223605517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09e0101b60 0x7f09e0112eb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f09d4009300 tx=0x7f09d4009e30 comp rx=0 tx=0).stop 2024-03-11T22:55:34.840 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.836+0000 7f09e6ec0700 1 -- 172.21.15.163:0/1223605517 shutdown_connections 2024-03-11T22:55:34.840 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.836+0000 7f09e6ec0700 1 --2- 172.21.15.163:0/1223605517 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f09d0036910 0x7f09d0038dc0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:34.840 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.836+0000 7f09e6ec0700 1 --2- 172.21.15.163:0/1223605517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09e0101b60 0x7f09e0112eb0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:34.841 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.836+0000 7f09e6ec0700 1 -- 172.21.15.163:0/1223605517 >> 172.21.15.163:0/1223605517 conn(0x7f09e0072850 msgr2=0x7f09e00ffe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:55:34.841 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.836+0000 7f09e6ec0700 1 -- 172.21.15.163:0/1223605517 shutdown_connections 2024-03-11T22:55:34.841 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:34.836+0000 7f09e6ec0700 1 -- 172.21.15.163:0/1223605517 wait complete. 2024-03-11T22:55:34.841 INFO:teuthology.orchestra.run.smithi163.stderr:dumped monmap epoch 1 2024-03-11T22:55:35.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:35 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:35.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:35 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:35.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:35 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi163.ivtqrh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:55:35.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:35 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi163.ivtqrh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T22:55:35.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:35 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:55:35.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:35 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:35.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:35 smithi153 ceph-mon[28962]: Deploying daemon mgr.smithi163.ivtqrh on smithi163 2024-03-11T22:55:35.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:35 smithi153 ceph-mon[28962]: from='client.? 172.21.15.163:0/1223605517' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:55:37.406 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-11T22:55:37.406 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:55:38.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:37 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:38.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:37 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:38.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:37 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:38.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:37 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:38.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:37 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:55:38.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:37 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:38.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:37 smithi153 ceph-mon[28962]: Deploying daemon mon.smithi163 on smithi163 2024-03-11T22:55:38.852 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:55:39.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:39 smithi153 ceph-mon[28962]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:39.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:39 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:41.035+0000 7f9f9cfad700 1 -- 172.21.15.163:0/3967581837 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f98102bd0 msgr2=0x7f9f98102fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:55:41.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:41.035+0000 7f9f9cfad700 1 --2- 172.21.15.163:0/3967581837 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f98102bd0 0x7f9f98102fa0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9f880048c0 tx=0x7f9f88009e30 comp rx=0 tx=0).stop 2024-03-11T22:55:41.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:41.035+0000 7f9f9cfad700 1 -- 172.21.15.163:0/3967581837 shutdown_connections 2024-03-11T22:55:41.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:41.035+0000 7f9f9cfad700 1 --2- 172.21.15.163:0/3967581837 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f98102bd0 0x7f9f98102fa0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:41.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:41.035+0000 7f9f9cfad700 1 -- 172.21.15.163:0/3967581837 >> 172.21.15.163:0/3967581837 conn(0x7f9f980f8760 msgr2=0x7f9f980fab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:55:41.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:41.035+0000 7f9f9cfad700 1 -- 172.21.15.163:0/3967581837 shutdown_connections 2024-03-11T22:55:41.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:41.035+0000 7f9f9cfad700 1 -- 172.21.15.163:0/3967581837 wait complete. 2024-03-11T22:55:41.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:41.035+0000 7f9f9cfad700 1 Processor -- start 2024-03-11T22:55:41.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:41.035+0000 7f9f9cfad700 1 -- start start 2024-03-11T22:55:41.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:41.036+0000 7f9f9cfad700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f98102bd0 0x7f9f98070290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:55:41.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:41.036+0000 7f9f9cfad700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f980707d0 con 0x7f9f98102bd0 2024-03-11T22:55:41.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:41.036+0000 7f9f9659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f98102bd0 0x7f9f98070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:55:41.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:41.036+0000 7f9f9659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f98102bd0 0x7f9f98070290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.163:45890/0 (socket says 172.21.15.163:45890) 2024-03-11T22:55:41.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:41.036+0000 7f9f9659c700 1 -- 172.21.15.163:0/31634237 learned_addr learned my addr 172.21.15.163:0/31634237 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-03-11T22:55:41.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:41.036+0000 7f9f9659c700 1 -- 172.21.15.163:0/31634237 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f88005040 con 0x7f9f98102bd0 2024-03-11T22:55:41.042 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:41.036+0000 7f9f9659c700 1 --2- 172.21.15.163:0/31634237 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f98102bd0 0x7f9f98070290 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9f88009330 tx=0x7f9f88009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:55:41.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:41.037+0000 7f9f86ffd700 1 -- 172.21.15.163:0/31634237 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f8800be50 con 0x7f9f98102bd0 2024-03-11T22:55:41.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:41.037+0000 7f9f86ffd700 1 -- 172.21.15.163:0/31634237 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9f88009630 con 0x7f9f98102bd0 2024-03-11T22:55:41.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:41.037+0000 7f9f9cfad700 1 -- 172.21.15.163:0/31634237 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f980709d0 con 0x7f9f98102bd0 2024-03-11T22:55:41.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:41.037+0000 7f9f86ffd700 1 -- 172.21.15.163:0/31634237 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f8801c3f0 con 0x7f9f98102bd0 2024-03-11T22:55:41.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:41.037+0000 7f9f9cfad700 1 -- 172.21.15.163:0/31634237 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f98070ec0 con 0x7f9f98102bd0 2024-03-11T22:55:41.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:41.038+0000 7f9f86ffd700 1 -- 172.21.15.163:0/31634237 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) v1 ==== 43483+0+0 (secure 0 0 0) 0x7f9f8801c660 con 0x7f9f98102bd0 2024-03-11T22:55:41.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:41.038+0000 7f9f86ffd700 1 --2- 172.21.15.163:0/31634237 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f9f800369b0 0x7f9f80038e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:55:41.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:41.038+0000 7f9f9cfad700 1 -- 172.21.15.163:0/31634237 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f98071100 con 0x7f9f98102bd0 2024-03-11T22:55:41.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:41.038+0000 7f9f86ffd700 1 -- 172.21.15.163:0/31634237 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f9f88049bf0 con 0x7f9f98102bd0 2024-03-11T22:55:41.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:41.038+0000 7f9f95d9b700 1 --2- 172.21.15.163:0/31634237 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f9f800369b0 0x7f9f80038e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:55:41.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:41.039+0000 7f9f95d9b700 1 --2- 172.21.15.163:0/31634237 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f9f800369b0 0x7f9f80038e60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c009990 tx=0x7f9f8c006e40 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:55:41.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:41.043+0000 7f9f86ffd700 1 -- 172.21.15.163:0/31634237 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f9f98071100 con 0x7f9f98102bd0 2024-03-11T22:55:41.189 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: mon.smithi163@-1(synchronizing).mds e1 new map 2024-03-11T22:55:41.189 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: mon.smithi163@-1(synchronizing).mds e1 print_map 2024-03-11T22:55:41.189 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout: e1 2024-03-11T22:55:41.190 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T22:55:41.190 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-11T22:55:41.190 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout: legacy client fscid: -1 2024-03-11T22:55:41.190 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout: 2024-03-11T22:55:41.190 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout: No filesystems configured 2024-03-11T22:55:41.190 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: mon.smithi163@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-11T22:55:41.190 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: mon.smithi163@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-11T22:55:41.191 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: mon.smithi163@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-11T22:55:41.191 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: mon.smithi163@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-11T22:55:41.191 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: mon.smithi163@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-11T22:55:41.191 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: mon.smithi163@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-11T22:55:41.191 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: mon.smithi163@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-03-11T22:55:41.191 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: mon.smithi163@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T22:55:41.191 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: mon.smithi163@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:55:41.191 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: mon.smithi163@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:55:41.191 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: mon.smithi163@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:55:41.191 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:41.192 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.192 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: Added host smithi163 2024-03-11T22:55:41.192 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:41.192 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.192 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:41.192 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:41.192 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.192 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.192 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.192 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.193 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.193 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.193 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T22:55:41.193 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T22:55:41.194 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.195 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: Deploying daemon grafana.smithi153 on smithi153 2024-03-11T22:55:41.195 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.195 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:55:41.195 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:41.195 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:41.195 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/655306157' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-11T22:55:41.195 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/655306157' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-11T22:55:41.195 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:55:41.196 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:41.196 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi153:172.21.15.153=smithi153;smithi163:172.21.15.163=smithi163", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:55:41.196 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: Saving service mon spec with placement smithi153:172.21.15.153=smithi153;smithi163:172.21.15.163=smithi163;count:2 2024-03-11T22:55:41.196 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.196 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:41.196 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:41.196 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='client.? 172.21.15.163:0/2674362688' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:55:41.196 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:41.196 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='client.? 172.21.15.163:0/3155790713' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:55:41.196 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:41.196 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.196 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.196 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.196 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.196 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.196 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.197 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: Deploying daemon prometheus.smithi153 on smithi153 2024-03-11T22:55:41.197 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:41.197 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.197 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='client.? 172.21.15.163:0/4142110104' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:55:41.197 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:41.197 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:41.197 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='client.? 172.21.15.163:0/103066825' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:55:41.197 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:41.197 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:41.197 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='client.? 172.21.15.163:0/2823203573' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:55:41.197 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.197 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.197 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.197 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T22:55:41.197 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14160 172.21.15.153:0/196858900' entity='mgr.smithi153.nyocep' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T22:55:41.197 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: mgrmap e14: smithi153.nyocep(active, since 54s) 2024-03-11T22:55:41.198 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='client.? 172.21.15.163:0/4072892998' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:55:41.198 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='client.? 172.21.15.163:0/3339061768' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:55:41.198 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: Active manager daemon smithi153.nyocep restarted 2024-03-11T22:55:41.198 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: Activating manager daemon smithi153.nyocep 2024-03-11T22:55:41.198 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: osdmap e5: 0 total, 0 up, 0 in 2024-03-11T22:55:41.198 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: mgrmap e15: smithi153.nyocep(active, starting, since 0.00297457s) 2024-03-11T22:55:41.198 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-03-11T22:55:41.198 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mgr metadata", "who": "smithi153.nyocep", "id": "smithi153.nyocep"}]: dispatch 2024-03-11T22:55:41.198 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:55:41.198 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:55:41.198 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:55:41.198 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: Manager daemon smithi153.nyocep is now available 2024-03-11T22:55:41.198 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:41.199 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:41.199 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.199 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:55:41.199 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.nyocep/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:55:41.199 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.nyocep/trash_purge_schedule"}]: dispatch 2024-03-11T22:55:41.199 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.199 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: mgrmap e16: smithi153.nyocep(active, since 1.00704s) 2024-03-11T22:55:41.199 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: [11/Mar/2024:22:55:19] ENGINE Bus STARTING 2024-03-11T22:55:41.199 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: [11/Mar/2024:22:55:19] ENGINE Serving on https://172.21.15.153:7150 2024-03-11T22:55:41.199 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: [11/Mar/2024:22:55:19] ENGINE Bus STARTED 2024-03-11T22:55:41.199 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.199 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.199 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.200 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='client.? 172.21.15.163:0/3179057404' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:55:41.200 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: mgrmap e17: smithi153.nyocep(active, since 2s) 2024-03-11T22:55:41.200 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.200 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.200 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.200 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.200 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config rm", "who": "osd/host:smithi153", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:55:41.200 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.200 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.200 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.200 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.200 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config rm", "who": "osd/host:smithi163", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:55:41.200 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:41.200 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:41.200 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-11T22:55:41.200 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: Updating smithi163:/etc/ceph/ceph.conf 2024-03-11T22:55:41.201 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: Updating smithi163:/var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:55:41.201 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: Updating smithi153:/var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:55:41.201 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: Updating smithi163:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:55:41.201 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:55:41.201 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.201 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.201 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.201 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.201 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.201 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi163", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-11T22:55:41.201 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi163", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-11T22:55:41.201 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:41.201 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: Updating smithi163:/var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:55:41.201 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: Updating smithi153:/var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:55:41.201 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: Deploying daemon crash.smithi163 on smithi163 2024-03-11T22:55:41.201 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='client.? 172.21.15.163:0/2672378778' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:55:41.201 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: Deploying daemon node-exporter.smithi163 on smithi163 2024-03-11T22:55:41.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='client.? 172.21.15.163:0/3214293718' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:55:41.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:55:41.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi163.ivtqrh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:55:41.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi163.ivtqrh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T22:55:41.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:55:41.202 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:41.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: Deploying daemon mgr.smithi163.ivtqrh on smithi163 2024-03-11T22:55:41.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='client.? 172.21.15.163:0/1223605517' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:55:41.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:55:41.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:41.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: Deploying daemon mon.smithi163 on smithi163 2024-03-11T22:55:41.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:41.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:41.203 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 ceph-mon[35584]: mon.smithi163@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-03-11T22:55:41.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:41.190+0000 7f9f86ffd700 1 -- 172.21.15.163:0/31634237 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9f88025400 con 0x7f9f98102bd0 2024-03-11T22:55:41.323 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:41.318+0000 7f9f9cfad700 1 -- 172.21.15.163:0/31634237 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9f98101ef0 con 0x7f9f98102bd0 2024-03-11T22:55:41.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 bash[35523]: fadb9d7bcbe86a0446b093bae1ebd5605a0f0900dab7e90940c4b9553d38c6f5 2024-03-11T22:55:41.489 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:41 smithi163 systemd[1]: Started Ceph mon.smithi163 for fa913b96-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:55:46.241 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:46.237+0000 7f9f86ffd700 1 -- 172.21.15.163:0/31634237 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1021 (secure 0 0 0) 0x7f9f88034090 con 0x7f9f98102bd0 2024-03-11T22:55:46.242 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-11T22:55:46.242 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":2,"fsid":"fa913b96-dff9-11ee-95c8-87774f69a715","modified":"2024-03-11T22:55:41.190593Z","created":"2024-03-11T22:53:11.271993Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi163","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-11T22:55:46.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:46.239+0000 7f9f9cfad700 1 -- 172.21.15.163:0/31634237 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f9f800369b0 msgr2=0x7f9f80038e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:55:46.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:46.240+0000 7f9f9cfad700 1 --2- 172.21.15.163:0/31634237 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f9f800369b0 0x7f9f80038e60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c009990 tx=0x7f9f8c006e40 comp rx=0 tx=0).stop 2024-03-11T22:55:46.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:46.240+0000 7f9f9cfad700 1 -- 172.21.15.163:0/31634237 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f98102bd0 msgr2=0x7f9f98070290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:55:46.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:46.240+0000 7f9f9cfad700 1 --2- 172.21.15.163:0/31634237 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f98102bd0 0x7f9f98070290 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9f88009330 tx=0x7f9f88009e30 comp rx=0 tx=0).stop 2024-03-11T22:55:46.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:46.240+0000 7f9f9cfad700 1 -- 172.21.15.163:0/31634237 shutdown_connections 2024-03-11T22:55:46.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:46.240+0000 7f9f9cfad700 1 --2- 172.21.15.163:0/31634237 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f9f800369b0 0x7f9f80038e60 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:46.244 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:46.240+0000 7f9f9cfad700 1 --2- 172.21.15.163:0/31634237 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9f98102bd0 0x7f9f98070290 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:46.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:46.240+0000 7f9f9cfad700 1 -- 172.21.15.163:0/31634237 >> 172.21.15.163:0/31634237 conn(0x7f9f980f8760 msgr2=0x7f9f980fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:55:46.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:46.240+0000 7f9f9cfad700 1 -- 172.21.15.163:0/31634237 shutdown_connections 2024-03-11T22:55:46.245 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:55:46.240+0000 7f9f9cfad700 1 -- 172.21.15.163:0/31634237 wait complete. 2024-03-11T22:55:46.245 INFO:teuthology.orchestra.run.smithi163.stderr:dumped monmap epoch 2 2024-03-11T22:55:46.569 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:46 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-03-11T22:55:46.569 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:46 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mon metadata", "id": "smithi163"}]: dispatch 2024-03-11T22:55:46.569 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:46 smithi163 ceph-mon[35584]: mon.smithi153 calling monitor election 2024-03-11T22:55:46.569 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:46 smithi163 ceph-mon[35584]: from='client.? 172.21.15.163:0/31634237' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:55:46.569 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:46 smithi163 ceph-mon[35584]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:46.569 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:46 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mon metadata", "id": "smithi163"}]: dispatch 2024-03-11T22:55:46.569 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:46 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mon metadata", "id": "smithi163"}]: dispatch 2024-03-11T22:55:46.569 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:46 smithi163 ceph-mon[35584]: mon.smithi163 calling monitor election 2024-03-11T22:55:46.569 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:46 smithi163 ceph-mon[35584]: from='mgr.? 172.21.15.163:0/1118446722' entity='mgr.smithi163.ivtqrh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi163.ivtqrh/crt"}]: dispatch 2024-03-11T22:55:46.569 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:46 smithi163 ceph-mon[35584]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:46.570 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:46 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mon metadata", "id": "smithi163"}]: dispatch 2024-03-11T22:55:46.570 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:46 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mon metadata", "id": "smithi163"}]: dispatch 2024-03-11T22:55:46.570 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:46 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mon metadata", "id": "smithi163"}]: dispatch 2024-03-11T22:55:46.570 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:46 smithi163 ceph-mon[35584]: mon.smithi153 is new leader, mons smithi153,smithi163 in quorum (ranks 0,1) 2024-03-11T22:55:46.570 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:46 smithi163 ceph-mon[35584]: monmap e2: 2 mons at {smithi153=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],smithi163=[v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:55:46.570 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:46 smithi163 ceph-mon[35584]: fsmap 2024-03-11T22:55:46.570 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:46 smithi163 ceph-mon[35584]: osdmap e5: 0 total, 0 up, 0 in 2024-03-11T22:55:46.570 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:46 smithi163 ceph-mon[35584]: mgrmap e17: smithi153.nyocep(active, since 28s) 2024-03-11T22:55:46.570 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:46 smithi163 ceph-mon[35584]: Standby manager daemon smithi163.ivtqrh started 2024-03-11T22:55:46.570 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:46 smithi163 ceph-mon[35584]: from='mgr.? 172.21.15.163:0/1118446722' entity='mgr.smithi163.ivtqrh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:55:46.570 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:46 smithi163 ceph-mon[35584]: overall HEALTH_OK 2024-03-11T22:55:46.570 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:46 smithi163 ceph-mon[35584]: from='mgr.? 172.21.15.163:0/1118446722' entity='mgr.smithi163.ivtqrh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi163.ivtqrh/key"}]: dispatch 2024-03-11T22:55:46.570 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:46 smithi163 ceph-mon[35584]: from='mgr.? 172.21.15.163:0/1118446722' entity='mgr.smithi163.ivtqrh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:55:46.570 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:46 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:46.571 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:46 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:46.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:46 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-03-11T22:55:46.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:46 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mon metadata", "id": "smithi163"}]: dispatch 2024-03-11T22:55:46.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:46 smithi153 ceph-mon[28962]: mon.smithi153 calling monitor election 2024-03-11T22:55:46.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:46 smithi153 ceph-mon[28962]: from='client.? 172.21.15.163:0/31634237' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:55:46.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:46 smithi153 ceph-mon[28962]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:46.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:46 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mon metadata", "id": "smithi163"}]: dispatch 2024-03-11T22:55:46.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:46 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mon metadata", "id": "smithi163"}]: dispatch 2024-03-11T22:55:46.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:46 smithi153 ceph-mon[28962]: mon.smithi163 calling monitor election 2024-03-11T22:55:46.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:46 smithi153 ceph-mon[28962]: from='mgr.? 172.21.15.163:0/1118446722' entity='mgr.smithi163.ivtqrh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi163.ivtqrh/crt"}]: dispatch 2024-03-11T22:55:46.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:46 smithi153 ceph-mon[28962]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:46.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:46 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mon metadata", "id": "smithi163"}]: dispatch 2024-03-11T22:55:46.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:46 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mon metadata", "id": "smithi163"}]: dispatch 2024-03-11T22:55:46.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:46 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mon metadata", "id": "smithi163"}]: dispatch 2024-03-11T22:55:46.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:46 smithi153 ceph-mon[28962]: mon.smithi153 is new leader, mons smithi153,smithi163 in quorum (ranks 0,1) 2024-03-11T22:55:46.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:46 smithi153 ceph-mon[28962]: monmap e2: 2 mons at {smithi153=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],smithi163=[v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:55:46.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:46 smithi153 ceph-mon[28962]: fsmap 2024-03-11T22:55:46.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:46 smithi153 ceph-mon[28962]: osdmap e5: 0 total, 0 up, 0 in 2024-03-11T22:55:46.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:46 smithi153 ceph-mon[28962]: mgrmap e17: smithi153.nyocep(active, since 28s) 2024-03-11T22:55:46.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:46 smithi153 ceph-mon[28962]: Standby manager daemon smithi163.ivtqrh started 2024-03-11T22:55:46.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:46 smithi153 ceph-mon[28962]: from='mgr.? 172.21.15.163:0/1118446722' entity='mgr.smithi163.ivtqrh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:55:46.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:46 smithi153 ceph-mon[28962]: overall HEALTH_OK 2024-03-11T22:55:46.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:46 smithi153 ceph-mon[28962]: from='mgr.? 172.21.15.163:0/1118446722' entity='mgr.smithi163.ivtqrh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi163.ivtqrh/key"}]: dispatch 2024-03-11T22:55:46.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:46 smithi153 ceph-mon[28962]: from='mgr.? 172.21.15.163:0/1118446722' entity='mgr.smithi163.ivtqrh' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:55:46.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:46 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:46.783 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:46 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:46.876 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-11T22:55:46.876 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph config generate-minimal-conf 2024-03-11T22:55:47.137 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:55:47.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:47 smithi153 ceph-mon[28962]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:47.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:47 smithi153 ceph-mon[28962]: mgrmap e18: smithi153.nyocep(active, since 28s), standbys: smithi163.ivtqrh 2024-03-11T22:55:47.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:47 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mgr metadata", "who": "smithi163.ivtqrh", "id": "smithi163.ivtqrh"}]: dispatch 2024-03-11T22:55:47.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:47 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:47.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:47 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:47.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:47 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:47.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:47 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:47.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:47 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mon metadata", "id": "smithi163"}]: dispatch 2024-03-11T22:55:47.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:47 smithi163 ceph-mon[35584]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:47.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:47 smithi163 ceph-mon[35584]: mgrmap e18: smithi153.nyocep(active, since 28s), standbys: smithi163.ivtqrh 2024-03-11T22:55:47.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:47 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mgr metadata", "who": "smithi163.ivtqrh", "id": "smithi163.ivtqrh"}]: dispatch 2024-03-11T22:55:47.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:47 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:47.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:47 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:47.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:47 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:47.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:47 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:47.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:47 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mon metadata", "id": "smithi163"}]: dispatch 2024-03-11T22:55:48.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.534+0000 7f54dce88700 1 -- 172.21.15.153:0/548946091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54d80fdc00 msgr2=0x7f54d80fdfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:55:48.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.534+0000 7f54dce88700 1 --2- 172.21.15.153:0/548946091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54d80fdc00 0x7f54d80fdfd0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f54c80048c0 tx=0x7f54c8009e30 comp rx=0 tx=0).stop 2024-03-11T22:55:48.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.534+0000 7f54dce88700 1 -- 172.21.15.153:0/548946091 shutdown_connections 2024-03-11T22:55:48.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.534+0000 7f54dce88700 1 --2- 172.21.15.153:0/548946091 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54d80fdc00 0x7f54d80fdfd0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:48.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.534+0000 7f54dce88700 1 -- 172.21.15.153:0/548946091 >> 172.21.15.153:0/548946091 conn(0x7f54d80f7660 msgr2=0x7f54d80f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:55:48.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.534+0000 7f54dce88700 1 -- 172.21.15.153:0/548946091 shutdown_connections 2024-03-11T22:55:48.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.534+0000 7f54dce88700 1 -- 172.21.15.153:0/548946091 wait complete. 2024-03-11T22:55:48.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.535+0000 7f54dce88700 1 Processor -- start 2024-03-11T22:55:48.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.535+0000 7f54dce88700 1 -- start start 2024-03-11T22:55:48.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.535+0000 7f54dce88700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54d80fdc00 0x7f54d8113140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:55:48.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.535+0000 7f54dce88700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54d8113680 0x7f54d810cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:55:48.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.535+0000 7f54dce88700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d8113c90 con 0x7f54d80fdc00 2024-03-11T22:55:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.535+0000 7f54dce88700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d810d520 con 0x7f54d8113680 2024-03-11T22:55:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.535+0000 7f54d659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54d80fdc00 0x7f54d8113140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:55:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.535+0000 7f54d659c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54d80fdc00 0x7f54d8113140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55280/0 (socket says 172.21.15.153:55280) 2024-03-11T22:55:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.535+0000 7f54d659c700 1 -- 172.21.15.153:0/3486824534 learned_addr learned my addr 172.21.15.153:0/3486824534 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:55:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.535+0000 7f54d5d9b700 1 --2- 172.21.15.153:0/3486824534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54d8113680 0x7f54d810cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:55:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.536+0000 7f54d5d9b700 1 -- 172.21.15.153:0/3486824534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54d8113680 msgr2=0x7f54d810cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-11T22:55:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.536+0000 7f54d5d9b700 1 -- 172.21.15.153:0/3486824534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54d8113680 msgr2=0x7f54d810cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-11T22:55:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.536+0000 7f54d659c700 1 -- 172.21.15.153:0/3486824534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54d8113680 msgr2=0x7f54d810cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:55:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.536+0000 7f54d5d9b700 1 --2- 172.21.15.153:0/3486824534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54d8113680 0x7f54d810cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-03-11T22:55:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.536+0000 7f54d5d9b700 1 --2- 172.21.15.153:0/3486824534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54d8113680 0x7f54d810cfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-11T22:55:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.536+0000 7f54d659c700 1 --2- 172.21.15.153:0/3486824534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54d8113680 0x7f54d810cfe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.536+0000 7f54d659c700 1 -- 172.21.15.153:0/3486824534 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c8005040 con 0x7f54d80fdc00 2024-03-11T22:55:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.536+0000 7f54d659c700 1 --2- 172.21.15.153:0/3486824534 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54d80fdc00 0x7f54d8113140 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f54c800b870 tx=0x7f54c8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:55:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.536+0000 7f54c6ffd700 1 -- 172.21.15.153:0/3486824534 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54c800e040 con 0x7f54d80fdc00 2024-03-11T22:55:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.536+0000 7f54dce88700 1 -- 172.21.15.153:0/3486824534 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54d810d7a0 con 0x7f54d80fdc00 2024-03-11T22:55:48.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.536+0000 7f54c6ffd700 1 -- 172.21.15.153:0/3486824534 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f54c8013420 con 0x7f54d80fdc00 2024-03-11T22:55:48.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.536+0000 7f54c6ffd700 1 -- 172.21.15.153:0/3486824534 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54c801c630 con 0x7f54d80fdc00 2024-03-11T22:55:48.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.536+0000 7f54dce88700 1 -- 172.21.15.153:0/3486824534 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54d810dce0 con 0x7f54d80fdc00 2024-03-11T22:55:48.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.537+0000 7f54dce88700 1 -- 172.21.15.153:0/3486824534 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54d8104430 con 0x7f54d80fdc00 2024-03-11T22:55:48.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.538+0000 7f54c6ffd700 1 -- 172.21.15.153:0/3486824534 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7f54c8013a20 con 0x7f54d80fdc00 2024-03-11T22:55:48.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.538+0000 7f54c6ffd700 1 --2- 172.21.15.153:0/3486824534 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f54c0069370 0x7f54c006b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:55:48.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.538+0000 7f54c6ffd700 1 -- 172.21.15.153:0/3486824534 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f54c8087430 con 0x7f54d80fdc00 2024-03-11T22:55:48.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.538+0000 7f54d5d9b700 1 --2- 172.21.15.153:0/3486824534 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f54c0069370 0x7f54c006b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:55:48.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.538+0000 7f54d5d9b700 1 --2- 172.21.15.153:0/3486824534 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f54c0069370 0x7f54c006b820 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f54cc006f30 tx=0x7f54cc008040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:55:48.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.542+0000 7f54c6ffd700 1 -- 172.21.15.153:0/3486824534 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f54c80582d0 con 0x7f54d80fdc00 2024-03-11T22:55:48.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:48 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:55:48.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:48 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:48.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:48 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:55:48.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:48 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:48.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.804+0000 7f54dce88700 1 -- 172.21.15.153:0/3486824534 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f54d8106f70 con 0x7f54d80fdc00 2024-03-11T22:55:48.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.804+0000 7f54c6ffd700 1 -- 172.21.15.153:0/3486824534 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+227 (secure 0 0 0) 0x7f54c8086be0 con 0x7f54d80fdc00 2024-03-11T22:55:48.809 INFO:teuthology.orchestra.run.smithi153.stdout:# minimal ceph.conf for fa913b96-dff9-11ee-95c8-87774f69a715 2024-03-11T22:55:48.809 INFO:teuthology.orchestra.run.smithi153.stdout:[global] 2024-03-11T22:55:48.809 INFO:teuthology.orchestra.run.smithi153.stdout: fsid = fa913b96-dff9-11ee-95c8-87774f69a715 2024-03-11T22:55:48.809 INFO:teuthology.orchestra.run.smithi153.stdout: mon_host = [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-03-11T22:55:48.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.807+0000 7f54dce88700 1 -- 172.21.15.153:0/3486824534 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f54c0069370 msgr2=0x7f54c006b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:55:48.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.807+0000 7f54dce88700 1 --2- 172.21.15.153:0/3486824534 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f54c0069370 0x7f54c006b820 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f54cc006f30 tx=0x7f54cc008040 comp rx=0 tx=0).stop 2024-03-11T22:55:48.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.807+0000 7f54dce88700 1 -- 172.21.15.153:0/3486824534 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54d80fdc00 msgr2=0x7f54d8113140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:55:48.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.807+0000 7f54dce88700 1 --2- 172.21.15.153:0/3486824534 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54d80fdc00 0x7f54d8113140 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f54c800b870 tx=0x7f54c8009e30 comp rx=0 tx=0).stop 2024-03-11T22:55:48.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.807+0000 7f54dce88700 1 -- 172.21.15.153:0/3486824534 shutdown_connections 2024-03-11T22:55:48.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.807+0000 7f54dce88700 1 --2- 172.21.15.153:0/3486824534 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54d80fdc00 0x7f54d8113140 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:48.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.807+0000 7f54dce88700 1 --2- 172.21.15.153:0/3486824534 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f54c0069370 0x7f54c006b820 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:48.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.807+0000 7f54dce88700 1 --2- 172.21.15.153:0/3486824534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54d8113680 0x7f54d810cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:48.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.807+0000 7f54dce88700 1 -- 172.21.15.153:0/3486824534 >> 172.21.15.153:0/3486824534 conn(0x7f54d80f7660 msgr2=0x7f54d81015b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:55:48.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.807+0000 7f54dce88700 1 -- 172.21.15.153:0/3486824534 shutdown_connections 2024-03-11T22:55:48.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:48.807+0000 7f54dce88700 1 -- 172.21.15.153:0/3486824534 wait complete. 2024-03-11T22:55:49.433 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-11T22:55:49.434 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-11T22:55:49.434 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T22:55:49.464 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-11T22:55:49.464 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:55:49.528 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:49 smithi153 ceph-mon[28962]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:49.528 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:49 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:49.528 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:49 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:49.528 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:49 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:49.528 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:49 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:49.528 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:49 smithi153 ceph-mon[28962]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-11T22:55:49.528 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:49 smithi153 ceph-mon[28962]: Updating smithi163:/etc/ceph/ceph.conf 2024-03-11T22:55:49.528 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:49 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/3486824534' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:49.528 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:49 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:49.528 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:49 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:49.529 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:49 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:49.529 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:49 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:49.529 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:49 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:49.529 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:49 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:55:49.529 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:49 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:55:49.529 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:49 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:49.532 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-11T22:55:49.532 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T22:55:49.561 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-11T22:55:49.561 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:55:49.629 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:49 smithi163 ceph-mon[35584]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:49.629 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:49 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:49.629 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:49 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:49.629 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:49 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:49.629 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:49 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:49.629 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:49 smithi163 ceph-mon[35584]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-11T22:55:49.629 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:49 smithi163 ceph-mon[35584]: Updating smithi163:/etc/ceph/ceph.conf 2024-03-11T22:55:49.629 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:49 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/3486824534' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:49.629 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:49 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:49.629 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:49 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:49.629 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:49 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:49.630 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:49 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:49.630 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:49 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:49.630 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:49 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:55:49.630 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:49 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:55:49.630 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:49 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:49.632 INFO:tasks.cephadm:Deploying OSDs... 2024-03-11T22:55:49.632 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-11T22:55:49.633 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T22:55:49.647 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T22:55:49.647 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_1 2024-03-11T22:55:49.701 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T22:55:49.701 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:55:49.702 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-11T22:55:49.702 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:55:49.702 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:55:49.702 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-11 22:55:09.932900519 +0000 2024-03-11T22:55:49.702 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-11 22:48:44.390572377 +0000 2024-03-11T22:55:49.702 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-11 22:48:44.390572377 +0000 2024-03-11T22:55:49.702 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-11 22:48:44.390572377 +0000 2024-03-11T22:55:49.702 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T22:55:49.767 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-11T22:55:49.767 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-11T22:55:49.767 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000192015 s, 2.7 MB/s 2024-03-11T22:55:49.768 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T22:55:49.823 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_2 2024-03-11T22:55:49.877 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T22:55:49.877 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:55:49.877 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-11T22:55:49.877 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:55:49.877 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:55:49.877 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-11 22:55:09.933900501 +0000 2024-03-11T22:55:49.878 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-11 22:48:44.727566517 +0000 2024-03-11T22:55:49.878 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-11 22:48:44.727566517 +0000 2024-03-11T22:55:49.878 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-11 22:48:44.727566517 +0000 2024-03-11T22:55:49.878 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T22:55:49.943 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-11T22:55:49.944 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-11T22:55:49.944 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000189728 s, 2.7 MB/s 2024-03-11T22:55:49.945 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T22:55:50.000 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_3 2024-03-11T22:55:50.054 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T22:55:50.054 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:55:50.055 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-11T22:55:50.055 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:55:50.055 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:55:50.055 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-11 22:55:09.933900501 +0000 2024-03-11T22:55:50.055 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-11 22:48:45.073560501 +0000 2024-03-11T22:55:50.055 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-11 22:48:45.073560501 +0000 2024-03-11T22:55:50.055 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-11 22:48:45.073560501 +0000 2024-03-11T22:55:50.055 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T22:55:50.120 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-11T22:55:50.120 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-11T22:55:50.120 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.00018273 s, 2.8 MB/s 2024-03-11T22:55:50.121 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T22:55:50.176 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_4 2024-03-11T22:55:50.232 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T22:55:50.232 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:55:50.232 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-11T22:55:50.232 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:55:50.232 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:55:50.232 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-11 22:55:09.934900483 +0000 2024-03-11T22:55:50.232 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-11 22:48:45.409554658 +0000 2024-03-11T22:55:50.232 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-11 22:48:45.409554658 +0000 2024-03-11T22:55:50.232 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-11 22:48:45.409554658 +0000 2024-03-11T22:55:50.232 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T22:55:50.299 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-11T22:55:50.300 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-11T22:55:50.300 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000296427 s, 1.7 MB/s 2024-03-11T22:55:50.301 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T22:55:50.357 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-11T22:55:50.357 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T22:55:50.376 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T22:55:50.376 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_1 2024-03-11T22:55:50.435 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T22:55:50.435 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:55:50.435 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-11T22:55:50.435 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:55:50.435 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:55:50.435 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-03-11 22:55:39.560213246 +0000 2024-03-11T22:55:50.435 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-03-11 22:48:45.175664869 +0000 2024-03-11T22:55:50.435 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-03-11 22:48:45.175664869 +0000 2024-03-11T22:55:50.435 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-03-11 22:48:45.175664869 +0000 2024-03-11T22:55:50.436 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T22:55:50.505 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-03-11T22:55:50.505 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-03-11T22:55:50.505 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000286703 s, 1.8 MB/s 2024-03-11T22:55:50.507 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T22:55:50.564 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_2 2024-03-11T22:55:50.621 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T22:55:50.621 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:55:50.621 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-11T22:55:50.621 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:55:50.621 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:55:50.621 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-03-11 22:55:39.561213222 +0000 2024-03-11T22:55:50.621 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-03-11 22:48:45.526656685 +0000 2024-03-11T22:55:50.622 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-03-11 22:48:45.526656685 +0000 2024-03-11T22:55:50.622 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-03-11 22:48:45.526656685 +0000 2024-03-11T22:55:50.622 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T22:55:50.678 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:50 smithi153 ceph-mon[28962]: Updating smithi163:/var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:55:50.678 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:50 smithi153 ceph-mon[28962]: Updating smithi153:/var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:55:50.678 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:50 smithi153 ceph-mon[28962]: Reconfiguring mon.smithi153 (unknown last config time)... 2024-03-11T22:55:50.679 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:50 smithi153 ceph-mon[28962]: Reconfiguring daemon mon.smithi153 on smithi153 2024-03-11T22:55:50.687 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:50 smithi163 ceph-mon[35584]: Updating smithi163:/var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:55:50.687 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:50 smithi163 ceph-mon[35584]: Updating smithi153:/var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:55:50.687 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:50 smithi163 ceph-mon[35584]: Reconfiguring mon.smithi153 (unknown last config time)... 2024-03-11T22:55:50.687 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:50 smithi163 ceph-mon[35584]: Reconfiguring daemon mon.smithi153 on smithi153 2024-03-11T22:55:50.689 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-03-11T22:55:50.689 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-03-11T22:55:50.689 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000208625 s, 2.5 MB/s 2024-03-11T22:55:50.690 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T22:55:50.747 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_3 2024-03-11T22:55:50.803 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T22:55:50.803 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:55:50.804 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-11T22:55:50.804 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:55:50.804 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:55:50.804 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-03-11 22:55:39.561213222 +0000 2024-03-11T22:55:50.804 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-03-11 22:48:45.883648362 +0000 2024-03-11T22:55:50.804 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-03-11 22:48:45.883648362 +0000 2024-03-11T22:55:50.804 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-03-11 22:48:45.883648362 +0000 2024-03-11T22:55:50.804 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T22:55:50.871 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-03-11T22:55:50.871 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-03-11T22:55:50.871 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000209549 s, 2.4 MB/s 2024-03-11T22:55:50.872 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T22:55:50.927 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_4 2024-03-11T22:55:50.983 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T22:55:50.983 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:55:50.983 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-11T22:55:50.983 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:55:50.983 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:55:50.984 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-03-11 22:55:39.561213222 +0000 2024-03-11T22:55:50.984 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-03-11 22:48:46.241640014 +0000 2024-03-11T22:55:50.984 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-03-11 22:48:46.241640014 +0000 2024-03-11T22:55:50.984 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-03-11 22:48:46.241640014 +0000 2024-03-11T22:55:50.984 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T22:55:51.049 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-03-11T22:55:51.050 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-03-11T22:55:51.050 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000193303 s, 2.6 MB/s 2024-03-11T22:55:51.051 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T22:55:51.107 INFO:tasks.cephadm:Deploying osd.0 on smithi153 with /dev/vg_nvme/lv_4... 2024-03-11T22:55:51.107 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-11T22:55:51.692 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:51 smithi153 ceph-mon[28962]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:51.692 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:51 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:51.692 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:51 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:51.693 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:51 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi153.nyocep", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:55:51.693 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:51 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:55:51.693 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:51 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:51.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:51 smithi163 ceph-mon[35584]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:51.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:51 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:51.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:51 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:51.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:51 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi153.nyocep", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:55:51.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:51 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:55:51.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:51 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:52.094 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:55:52.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:52 smithi163 ceph-mon[35584]: Reconfiguring mgr.smithi153.nyocep (unknown last config time)... 2024-03-11T22:55:52.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:52 smithi163 ceph-mon[35584]: Reconfiguring daemon mgr.smithi153.nyocep on smithi153 2024-03-11T22:55:52.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:52 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:52.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:52 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:52.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:52 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi153", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-11T22:55:52.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:52 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:52.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:52 smithi153 ceph-mon[28962]: Reconfiguring mgr.smithi153.nyocep (unknown last config time)... 2024-03-11T22:55:52.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:52 smithi153 ceph-mon[28962]: Reconfiguring daemon mgr.smithi153.nyocep on smithi153 2024-03-11T22:55:52.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:52 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:52.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:52 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:52.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:52 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi153", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-11T22:55:52.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:52 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:53.731 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:53 smithi153 ceph-mon[28962]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:53.731 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:53 smithi153 ceph-mon[28962]: Reconfiguring crash.smithi153 (monmap changed)... 2024-03-11T22:55:53.731 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:53 smithi153 ceph-mon[28962]: Reconfiguring daemon crash.smithi153 on smithi153 2024-03-11T22:55:53.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:53 smithi163 ceph-mon[35584]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:53.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:53 smithi163 ceph-mon[35584]: Reconfiguring crash.smithi153 (monmap changed)... 2024-03-11T22:55:53.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:53 smithi163 ceph-mon[35584]: Reconfiguring daemon crash.smithi153 on smithi153 2024-03-11T22:55:54.781 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-11T22:55:54.781 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-11T22:55:54.781 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-03-11T22:55:54.781 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-03-11T22:55:54.781 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0117071 s, 896 MB/s 2024-03-11T22:55:54.781 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-03-11T22:55:55.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:55 smithi153 ceph-mon[28962]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:55.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:55 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:55.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:55 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:55.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:55 smithi153 ceph-mon[28962]: Reconfiguring alertmanager.smithi153 (dependencies changed)... 2024-03-11T22:55:55.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:55 smithi153 ceph-mon[28962]: Reconfiguring daemon alertmanager.smithi153 on smithi153 2024-03-11T22:55:55.562 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi153:vg_nvme/lv_4 2024-03-11T22:55:55.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:55 smithi163 ceph-mon[35584]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:55.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:55 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:55.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:55 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:55:55.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:55 smithi163 ceph-mon[35584]: Reconfiguring alertmanager.smithi153 (dependencies changed)... 2024-03-11T22:55:55.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:55 smithi163 ceph-mon[35584]: Reconfiguring daemon alertmanager.smithi153 on smithi153 2024-03-11T22:55:57.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:57 smithi153 ceph-mon[28962]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:57.540 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:57 smithi163 ceph-mon[35584]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:57.763 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:55:59.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:59.449+0000 7f58215c7700 1 -- 172.21.15.153:0/177249963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f581c0fbe50 msgr2=0x7f581c0fc260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:55:59.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:59.449+0000 7f58215c7700 1 --2- 172.21.15.153:0/177249963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f581c0fbe50 0x7f581c0fc260 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f580c004940 tx=0x7f580c009e30 comp rx=0 tx=0).stop 2024-03-11T22:55:59.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:59.449+0000 7f58215c7700 1 -- 172.21.15.153:0/177249963 shutdown_connections 2024-03-11T22:55:59.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:59.449+0000 7f58215c7700 1 --2- 172.21.15.153:0/177249963 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f581c0fcfc0 0x7f581c0fd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:59.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:59.449+0000 7f58215c7700 1 --2- 172.21.15.153:0/177249963 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f581c0fbe50 0x7f581c0fc260 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:59.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:59.449+0000 7f58215c7700 1 -- 172.21.15.153:0/177249963 >> 172.21.15.153:0/177249963 conn(0x7f581c0f7510 msgr2=0x7f581c0f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:55:59.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:59.449+0000 7f58215c7700 1 -- 172.21.15.153:0/177249963 shutdown_connections 2024-03-11T22:55:59.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:59.449+0000 7f58215c7700 1 -- 172.21.15.153:0/177249963 wait complete. 2024-03-11T22:55:59.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:59.449+0000 7f58215c7700 1 Processor -- start 2024-03-11T22:55:59.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:59.449+0000 7f58215c7700 1 -- start start 2024-03-11T22:55:59.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:59.450+0000 7f58215c7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f581c0fbe50 0x7f581c10c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:55:59.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:59.450+0000 7f58215c7700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f581c0fcfc0 0x7f581c10cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:55:59.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:59.450+0000 7f58215c7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f581c10d390 con 0x7f581c0fbe50 2024-03-11T22:55:59.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:59.450+0000 7f58215c7700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f581c10dc30 con 0x7f581c0fcfc0 2024-03-11T22:55:59.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:59.450+0000 7f581affd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f581c0fbe50 0x7f581c10c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:55:59.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:59.450+0000 7f581affd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f581c0fbe50 0x7f581c10c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57562/0 (socket says 172.21.15.153:57562) 2024-03-11T22:55:59.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:59.450+0000 7f581affd700 1 -- 172.21.15.153:0/3486753470 learned_addr learned my addr 172.21.15.153:0/3486753470 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:55:59.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:59.450+0000 7f58127fc700 1 --2- 172.21.15.153:0/3486753470 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f581c0fcfc0 0x7f581c10cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:55:59.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:59.450+0000 7f581affd700 1 -- 172.21.15.153:0/3486753470 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f581c0fcfc0 msgr2=0x7f581c10cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:55:59.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:59.450+0000 7f581affd700 1 --2- 172.21.15.153:0/3486753470 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f581c0fcfc0 0x7f581c10cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:55:59.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:59.450+0000 7f581affd700 1 -- 172.21.15.153:0/3486753470 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f580c005040 con 0x7f581c0fbe50 2024-03-11T22:55:59.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:59.450+0000 7f581affd700 1 --2- 172.21.15.153:0/3486753470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f581c0fbe50 0x7f581c10c850 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f580c009fd0 tx=0x7f580c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:55:59.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:59.450+0000 7f5813fff700 1 -- 172.21.15.153:0/3486753470 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f580c00e040 con 0x7f581c0fbe50 2024-03-11T22:55:59.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:59.450+0000 7f5813fff700 1 -- 172.21.15.153:0/3486753470 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f580c00bcc0 con 0x7f581c0fbe50 2024-03-11T22:55:59.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:59.450+0000 7f58215c7700 1 -- 172.21.15.153:0/3486753470 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f581c10deb0 con 0x7f581c0fbe50 2024-03-11T22:55:59.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:59.450+0000 7f5813fff700 1 -- 172.21.15.153:0/3486753470 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f580c01c630 con 0x7f581c0fbe50 2024-03-11T22:55:59.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:59.450+0000 7f58215c7700 1 -- 172.21.15.153:0/3486753470 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f581c10e3f0 con 0x7f581c0fbe50 2024-03-11T22:55:59.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:59.451+0000 7f58215c7700 1 -- 172.21.15.153:0/3486753470 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f581c04bed0 con 0x7f581c0fbe50 2024-03-11T22:55:59.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:59.452+0000 7f5813fff700 1 -- 172.21.15.153:0/3486753470 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7f580c013980 con 0x7f581c0fbe50 2024-03-11T22:55:59.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:59.452+0000 7f5813fff700 1 --2- 172.21.15.153:0/3486753470 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f5808069210 0x7f580806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:55:59.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:59.452+0000 7f5813fff700 1 -- 172.21.15.153:0/3486753470 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f580c087540 con 0x7f581c0fbe50 2024-03-11T22:55:59.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:59.452+0000 7f58127fc700 1 --2- 172.21.15.153:0/3486753470 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f5808069210 0x7f580806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:55:59.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:59.453+0000 7f58127fc700 1 --2- 172.21.15.153:0/3486753470 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f5808069210 0x7f580806b6c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5804004990 tx=0x7f5804009ee0 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:55:59.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:59.458+0000 7f5813fff700 1 -- 172.21.15.153:0/3486753470 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f580c0583e0 con 0x7f581c0fbe50 2024-03-11T22:55:59.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:55:59 smithi153 ceph-mon[28962]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:59.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:55:59.653+0000 7f58215c7700 1 -- 172.21.15.153:0/3486753470 --> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f581c10ea60 con 0x7f5808069210 2024-03-11T22:55:59.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:55:59 smithi163 ceph-mon[35584]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:56:00.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:00 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:56:00.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:00 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:56:00.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:00 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:00.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:00 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:56:00.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:00 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:56:00.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:00 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:01.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:01 smithi153 ceph-mon[28962]: from='client.14234 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:56:01.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:01 smithi153 ceph-mon[28962]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:56:01.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:01 smithi163 ceph-mon[35584]: from='client.14234 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:56:01.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:01 smithi163 ceph-mon[35584]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:56:03.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:03 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:03.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:03 smithi163 ceph-mon[35584]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:56:03.489 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:03 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:03.489 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:03 smithi163 ceph-mon[35584]: Reconfiguring grafana.smithi153 (dependencies changed)... 2024-03-11T22:56:03.489 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:03 smithi163 ceph-mon[35584]: Reconfiguring daemon grafana.smithi153 on smithi153 2024-03-11T22:56:03.513 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:03 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:03.514 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:03 smithi153 ceph-mon[28962]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:56:03.514 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:03 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:03.514 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:03 smithi153 ceph-mon[28962]: Reconfiguring grafana.smithi153 (dependencies changed)... 2024-03-11T22:56:03.514 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:03 smithi153 ceph-mon[28962]: Reconfiguring daemon grafana.smithi153 on smithi153 2024-03-11T22:56:04.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:04 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:56:04.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:04 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/3611890395' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "542731eb-3fb3-450b-93d6-c6a8a9df1659"}]: dispatch 2024-03-11T22:56:04.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:04 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/3611890395' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "542731eb-3fb3-450b-93d6-c6a8a9df1659"}]': finished 2024-03-11T22:56:04.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:04 smithi153 ceph-mon[28962]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T22:56:04.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:04 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:56:04.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:04 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/3181028157' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:56:04.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:04 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:56:04.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:04 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/3611890395' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "542731eb-3fb3-450b-93d6-c6a8a9df1659"}]: dispatch 2024-03-11T22:56:04.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:04 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/3611890395' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "542731eb-3fb3-450b-93d6-c6a8a9df1659"}]': finished 2024-03-11T22:56:04.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:04 smithi163 ceph-mon[35584]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T22:56:04.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:04 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:56:04.489 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:04 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/3181028157' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:56:05.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:05 smithi153 ceph-mon[28962]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:56:05.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:05 smithi163 ceph-mon[35584]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:56:07.352 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:07 smithi153 ceph-mon[28962]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:56:07.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:07 smithi163 ceph-mon[35584]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:56:09.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:09 smithi153 ceph-mon[28962]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:56:09.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:09 smithi163 ceph-mon[35584]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:56:11.518 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:11 smithi153 ceph-mon[28962]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:56:11.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:11 smithi163 ceph-mon[35584]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:56:13.627 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:13 smithi153 ceph-mon[28962]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:56:13.628 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:13 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:13.628 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:13 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:13.628 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:13 smithi153 ceph-mon[28962]: Reconfiguring prometheus.smithi153 (dependencies changed)... 2024-03-11T22:56:13.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:13 smithi163 ceph-mon[35584]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:56:13.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:13 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:13.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:13 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:13.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:13 smithi163 ceph-mon[35584]: Reconfiguring prometheus.smithi153 (dependencies changed)... 2024-03-11T22:56:14.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:14 smithi153 ceph-mon[28962]: Reconfiguring daemon prometheus.smithi153 on smithi153 2024-03-11T22:56:14.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:14 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T22:56:14.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:14 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:14.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:14 smithi163 ceph-mon[35584]: Reconfiguring daemon prometheus.smithi153 on smithi153 2024-03-11T22:56:14.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:14 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T22:56:14.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:14 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:15.742 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:15 smithi153 ceph-mon[28962]: Deploying daemon osd.0 on smithi153 2024-03-11T22:56:15.742 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:15 smithi153 ceph-mon[28962]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:56:15.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:15 smithi163 ceph-mon[35584]: Deploying daemon osd.0 on smithi153 2024-03-11T22:56:15.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:15 smithi163 ceph-mon[35584]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:56:17.905 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:17 smithi153 ceph-mon[28962]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:56:17.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:17 smithi163 ceph-mon[35584]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:56:18.776 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:18 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:56:18.776 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:18 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:18.776 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:18 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:18.776 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:18 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi163", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-11T22:56:18.776 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:18 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:18.891 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:18 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:56:18.891 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:18 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:18.891 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:18 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:18.891 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:18 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi163", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-11T22:56:18.891 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:18 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:19.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:19 smithi153 ceph-mon[28962]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:56:19.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:19 smithi153 ceph-mon[28962]: Reconfiguring crash.smithi163 (monmap changed)... 2024-03-11T22:56:19.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:19 smithi153 ceph-mon[28962]: Reconfiguring daemon crash.smithi163 on smithi163 2024-03-11T22:56:19.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:19 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:19.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:19 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:19.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:19 smithi153 ceph-mon[28962]: Reconfiguring mgr.smithi163.ivtqrh (monmap changed)... 2024-03-11T22:56:19.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:19 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi163.ivtqrh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:56:19.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:19 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:56:19.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:19 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:19.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:19 smithi153 ceph-mon[28962]: Reconfiguring daemon mgr.smithi163.ivtqrh on smithi163 2024-03-11T22:56:19.912 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:19 smithi163 ceph-mon[35584]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:56:19.912 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:19 smithi163 ceph-mon[35584]: Reconfiguring crash.smithi163 (monmap changed)... 2024-03-11T22:56:19.912 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:19 smithi163 ceph-mon[35584]: Reconfiguring daemon crash.smithi163 on smithi163 2024-03-11T22:56:19.912 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:19 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:19.912 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:19 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:19.912 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:19 smithi163 ceph-mon[35584]: Reconfiguring mgr.smithi163.ivtqrh (monmap changed)... 2024-03-11T22:56:19.913 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:19 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi163.ivtqrh", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:56:19.913 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:19 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:56:19.913 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:19 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:19.913 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:19 smithi163 ceph-mon[35584]: Reconfiguring daemon mgr.smithi163.ivtqrh on smithi163 2024-03-11T22:56:20.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:20 smithi153 ceph-mon[28962]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:56:20.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:20 smithi163 ceph-mon[35584]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:56:21.916 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:21 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:21.916 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:21 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:21.916 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:21 smithi163 ceph-mon[35584]: Reconfiguring mon.smithi163 (monmap changed)... 2024-03-11T22:56:21.916 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:21 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:56:21.916 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:21 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:56:21.916 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:21 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:21.916 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:21 smithi163 ceph-mon[35584]: Reconfiguring daemon mon.smithi163 on smithi163 2024-03-11T22:56:21.998 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:21 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:21.998 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:21 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:21.998 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:21 smithi153 ceph-mon[28962]: Reconfiguring mon.smithi163 (monmap changed)... 2024-03-11T22:56:21.998 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:21 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:56:21.998 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:21 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:56:21.999 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:21 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:21.999 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:21 smithi153 ceph-mon[28962]: Reconfiguring daemon mon.smithi163 on smithi163 2024-03-11T22:56:23.238 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:22 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:23.239 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:22 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:23.239 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:22 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T22:56:23.239 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:22 smithi163 ceph-mon[35584]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T22:56:23.239 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:22 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi153.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T22:56:23.239 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:22 smithi163 ceph-mon[35584]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi153.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T22:56:23.239 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:22 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:23.239 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:22 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T22:56:23.239 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:22 smithi163 ceph-mon[35584]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T22:56:23.239 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:22 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi153.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T22:56:23.239 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:22 smithi163 ceph-mon[35584]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi153.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T22:56:23.239 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:22 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:23.240 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:22 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T22:56:23.240 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:22 smithi163 ceph-mon[35584]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T22:56:23.240 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:22 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi153.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T22:56:23.240 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:22 smithi163 ceph-mon[35584]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi153.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T22:56:23.240 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:22 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:23.240 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:22 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:56:23.240 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:22 smithi163 ceph-mon[35584]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:56:23.240 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:22 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:23.240 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:22 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:23.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:22 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:23.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:22 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:23.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:22 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T22:56:23.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:22 smithi153 ceph-mon[28962]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T22:56:23.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:22 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi153.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T22:56:23.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:22 smithi153 ceph-mon[28962]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi153.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T22:56:23.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:22 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:23.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:22 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T22:56:23.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:22 smithi153 ceph-mon[28962]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T22:56:23.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:22 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi153.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T22:56:23.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:22 smithi153 ceph-mon[28962]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi153.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T22:56:23.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:22 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:23.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:22 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T22:56:23.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:22 smithi153 ceph-mon[28962]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T22:56:23.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:22 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi153.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T22:56:23.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:22 smithi153 ceph-mon[28962]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi153.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T22:56:23.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:22 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:23.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:22 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:56:23.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:22 smithi153 ceph-mon[28962]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:56:23.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:22 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:23.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:22 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:25.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:24 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:25.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:24 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:25.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:24 smithi153 ceph-mon[28962]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:56:25.238 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:24 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:25.238 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:24 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:25.238 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:24 smithi163 ceph-mon[35584]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:56:26.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:26.627+0000 7f5813fff700 1 -- 172.21.15.153:0/3486753470 <== mgr.14204 v2:172.21.15.153:6800/29938766 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f581c10ea60 con 0x7f5808069210 2024-03-11T22:56:26.633 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 0 on host 'smithi153' 2024-03-11T22:56:26.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:26.629+0000 7f58215c7700 1 -- 172.21.15.153:0/3486753470 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f5808069210 msgr2=0x7f580806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:56:26.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:26.629+0000 7f58215c7700 1 --2- 172.21.15.153:0/3486753470 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f5808069210 0x7f580806b6c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5804004990 tx=0x7f5804009ee0 comp rx=0 tx=0).stop 2024-03-11T22:56:26.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:26.629+0000 7f58215c7700 1 -- 172.21.15.153:0/3486753470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f581c0fbe50 msgr2=0x7f581c10c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:56:26.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:26.629+0000 7f58215c7700 1 --2- 172.21.15.153:0/3486753470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f581c0fbe50 0x7f581c10c850 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f580c009fd0 tx=0x7f580c009e30 comp rx=0 tx=0).stop 2024-03-11T22:56:26.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:26.630+0000 7f58215c7700 1 -- 172.21.15.153:0/3486753470 shutdown_connections 2024-03-11T22:56:26.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:26.630+0000 7f58215c7700 1 --2- 172.21.15.153:0/3486753470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f581c0fbe50 0x7f581c10c850 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:56:26.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:26.630+0000 7f58215c7700 1 --2- 172.21.15.153:0/3486753470 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f5808069210 0x7f580806b6c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:56:26.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:26.630+0000 7f58215c7700 1 --2- 172.21.15.153:0/3486753470 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f581c0fcfc0 0x7f581c10cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:56:26.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:26.630+0000 7f58215c7700 1 -- 172.21.15.153:0/3486753470 >> 172.21.15.153:0/3486753470 conn(0x7f581c0f7510 msgr2=0x7f581c100280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:56:26.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:26.630+0000 7f58215c7700 1 -- 172.21.15.153:0/3486753470 shutdown_connections 2024-03-11T22:56:26.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:26.630+0000 7f58215c7700 1 -- 172.21.15.153:0/3486753470 wait complete. 2024-03-11T22:56:27.372 DEBUG:teuthology.orchestra.run.smithi153:osd.0> sudo journalctl -f -n 0 -u ceph-fa913b96-dff9-11ee-95c8-87774f69a715@osd.0.service 2024-03-11T22:56:27.374 INFO:tasks.cephadm:Deploying osd.1 on smithi153 with /dev/vg_nvme/lv_3... 2024-03-11T22:56:27.374 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-11T22:56:27.756 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:56:27.778 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:27 smithi153 ceph-mon[28962]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:56:27.778 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:27 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:27.778 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:27 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:27.778 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:27 smithi153 ceph-mon[28962]: from='osd.0 [v2:172.21.15.153:6802/1406194697,v1:172.21.15.153:6803/1406194697]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T22:56:27.942 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:27 smithi163 ceph-mon[35584]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:56:27.942 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:27 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:27.942 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:27 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:27.942 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:27 smithi163 ceph-mon[35584]: from='osd.0 [v2:172.21.15.153:6802/1406194697,v1:172.21.15.153:6803/1406194697]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T22:56:28.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:28 smithi153 ceph-mon[28962]: from='osd.0 [v2:172.21.15.153:6802/1406194697,v1:172.21.15.153:6803/1406194697]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T22:56:28.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:28 smithi153 ceph-mon[28962]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T22:56:28.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:28 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:56:28.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:28 smithi153 ceph-mon[28962]: from='osd.0 [v2:172.21.15.153:6802/1406194697,v1:172.21.15.153:6803/1406194697]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-11T22:56:28.782 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 22:56:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T22:56:28.595+0000 7f3923956700 -1 osd.0 0 waiting for initial osdmap 2024-03-11T22:56:28.782 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 22:56:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T22:56:28.598+0000 7f391d74f700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:56:28.960 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:28 smithi163 ceph-mon[35584]: from='osd.0 [v2:172.21.15.153:6802/1406194697,v1:172.21.15.153:6803/1406194697]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T22:56:28.960 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:28 smithi163 ceph-mon[35584]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T22:56:28.960 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:28 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:56:28.960 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:28 smithi163 ceph-mon[35584]: from='osd.0 [v2:172.21.15.153:6802/1406194697,v1:172.21.15.153:6803/1406194697]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-11T22:56:29.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:29 smithi153 ceph-mon[28962]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:56:29.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:29 smithi153 ceph-mon[28962]: from='osd.0 [v2:172.21.15.153:6802/1406194697,v1:172.21.15.153:6803/1406194697]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-11T22:56:29.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:29 smithi153 ceph-mon[28962]: osdmap e8: 1 total, 0 up, 1 in 2024-03-11T22:56:29.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:29 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:56:29.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:29 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:56:29.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:29 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:29.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:29 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:29.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:29 smithi163 ceph-mon[35584]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:56:29.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:29 smithi163 ceph-mon[35584]: from='osd.0 [v2:172.21.15.153:6802/1406194697,v1:172.21.15.153:6803/1406194697]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-11T22:56:29.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:29 smithi163 ceph-mon[35584]: osdmap e8: 1 total, 0 up, 1 in 2024-03-11T22:56:29.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:29 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:56:29.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:29 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:56:29.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:29 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:29.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:29 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:30.737 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:30 smithi153 ceph-mon[28962]: purged_snaps scrub starts 2024-03-11T22:56:30.737 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:30 smithi153 ceph-mon[28962]: purged_snaps scrub ok 2024-03-11T22:56:30.737 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:30 smithi153 ceph-mon[28962]: osd.0 [v2:172.21.15.153:6802/1406194697,v1:172.21.15.153:6803/1406194697] boot 2024-03-11T22:56:30.737 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:30 smithi153 ceph-mon[28962]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T22:56:30.738 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:30 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:56:30.789 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-11T22:56:30.790 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-11T22:56:30.790 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-03-11T22:56:30.790 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-03-11T22:56:30.790 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0131339 s, 798 MB/s 2024-03-11T22:56:30.790 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-03-11T22:56:30.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:30 smithi163 ceph-mon[35584]: purged_snaps scrub starts 2024-03-11T22:56:30.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:30 smithi163 ceph-mon[35584]: purged_snaps scrub ok 2024-03-11T22:56:30.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:30 smithi163 ceph-mon[35584]: osd.0 [v2:172.21.15.153:6802/1406194697,v1:172.21.15.153:6803/1406194697] boot 2024-03-11T22:56:30.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:30 smithi163 ceph-mon[35584]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T22:56:30.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:30 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:56:31.863 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:31 smithi153 ceph-mon[28962]: pgmap v33: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:31.863 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:31 smithi153 ceph-mon[28962]: osdmap e10: 1 total, 1 up, 1 in 2024-03-11T22:56:31.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:31 smithi163 ceph-mon[35584]: pgmap v33: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:31.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:31 smithi163 ceph-mon[35584]: osdmap e10: 1 total, 1 up, 1 in 2024-03-11T22:56:32.274 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi153:vg_nvme/lv_3 2024-03-11T22:56:32.762 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:56:32.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:32 smithi153 ceph-mon[28962]: pgmap v35: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:32.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:32 smithi163 ceph-mon[35584]: pgmap v35: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:33.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:33 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:56:34.010 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:33 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:56:34.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:34.908+0000 7f9969ebb700 1 -- 172.21.15.153:0/2027321048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9964071f10 msgr2=0x7f9964070380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:56:34.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:34.908+0000 7f9969ebb700 1 --2- 172.21.15.153:0/2027321048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9964071f10 0x7f9964070380 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9954004940 tx=0x7f9954009e30 comp rx=0 tx=0).stop 2024-03-11T22:56:34.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:34.909+0000 7f9969ebb700 1 -- 172.21.15.153:0/2027321048 shutdown_connections 2024-03-11T22:56:34.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:34.909+0000 7f9969ebb700 1 --2- 172.21.15.153:0/2027321048 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99640708c0 0x7f9964070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:56:34.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:34.909+0000 7f9969ebb700 1 --2- 172.21.15.153:0/2027321048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9964071f10 0x7f9964070380 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:56:34.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:34.909+0000 7f9969ebb700 1 -- 172.21.15.153:0/2027321048 >> 172.21.15.153:0/2027321048 conn(0x7f99640f8620 msgr2=0x7f99640faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:56:34.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:34.909+0000 7f9969ebb700 1 -- 172.21.15.153:0/2027321048 shutdown_connections 2024-03-11T22:56:34.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:34.909+0000 7f9969ebb700 1 -- 172.21.15.153:0/2027321048 wait complete. 2024-03-11T22:56:34.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:34.909+0000 7f9969ebb700 1 Processor -- start 2024-03-11T22:56:34.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:34.909+0000 7f9969ebb700 1 -- start start 2024-03-11T22:56:34.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:34.909+0000 7f9969ebb700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99640708c0 0x7f9964113f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:56:34.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:34.909+0000 7f9969ebb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9964071f10 0x7f9964114470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:56:34.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:34.909+0000 7f9969ebb700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9964114a70 con 0x7f9964071f10 2024-03-11T22:56:34.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:34.909+0000 7f9969ebb700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9964115310 con 0x7f99640708c0 2024-03-11T22:56:34.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:34.910+0000 7f9962ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9964071f10 0x7f9964114470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:56:34.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:34.910+0000 7f9962ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9964071f10 0x7f9964114470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56056/0 (socket says 172.21.15.153:56056) 2024-03-11T22:56:34.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:34.910+0000 7f9962ffd700 1 -- 172.21.15.153:0/762687155 learned_addr learned my addr 172.21.15.153:0/762687155 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:56:34.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:34.910+0000 7f99637fe700 1 --2- 172.21.15.153:0/762687155 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99640708c0 0x7f9964113f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:56:34.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:34.910+0000 7f9962ffd700 1 -- 172.21.15.153:0/762687155 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99640708c0 msgr2=0x7f9964113f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:56:34.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:34.910+0000 7f9962ffd700 1 --2- 172.21.15.153:0/762687155 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99640708c0 0x7f9964113f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:56:34.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:34.910+0000 7f9962ffd700 1 -- 172.21.15.153:0/762687155 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9954005040 con 0x7f9964071f10 2024-03-11T22:56:34.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:34.910+0000 7f9962ffd700 1 --2- 172.21.15.153:0/762687155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9964071f10 0x7f9964114470 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9958002a50 tx=0x7f99580046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:56:34.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:34.910+0000 7f994bfff700 1 -- 172.21.15.153:0/762687155 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9958007cc0 con 0x7f9964071f10 2024-03-11T22:56:34.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:34.910+0000 7f994bfff700 1 -- 172.21.15.153:0/762687155 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f995800f040 con 0x7f9964071f10 2024-03-11T22:56:34.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:34.910+0000 7f994bfff700 1 -- 172.21.15.153:0/762687155 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f995800da30 con 0x7f9964071f10 2024-03-11T22:56:34.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:34.910+0000 7f9969ebb700 1 -- 172.21.15.153:0/762687155 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99641155f0 con 0x7f9964071f10 2024-03-11T22:56:34.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:34.910+0000 7f9969ebb700 1 -- 172.21.15.153:0/762687155 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9964115b30 con 0x7f9964071f10 2024-03-11T22:56:34.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:34.911+0000 7f9969ebb700 1 -- 172.21.15.153:0/762687155 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f996404bed0 con 0x7f9964071f10 2024-03-11T22:56:34.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:34.912+0000 7f994bfff700 1 -- 172.21.15.153:0/762687155 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7f995800db90 con 0x7f9964071f10 2024-03-11T22:56:34.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:34.912+0000 7f994bfff700 1 --2- 172.21.15.153:0/762687155 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f994c069210 0x7f994c06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:56:34.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:34.912+0000 7f994bfff700 1 -- 172.21.15.153:0/762687155 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1978+0+0 (secure 0 0 0) 0x7f9958085fb0 con 0x7f9964071f10 2024-03-11T22:56:34.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:34.912+0000 7f99637fe700 1 --2- 172.21.15.153:0/762687155 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f994c069210 0x7f994c06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:56:34.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:34.912+0000 7f99637fe700 1 --2- 172.21.15.153:0/762687155 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f994c069210 0x7f994c06b6c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9954004f90 tx=0x7f9954009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:56:34.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:34.916+0000 7f994bfff700 1 -- 172.21.15.153:0/762687155 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f9958056bf0 con 0x7f9964071f10 2024-03-11T22:56:35.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:35.123+0000 7f9969ebb700 1 -- 172.21.15.153:0/762687155 --> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f9964115fd0 con 0x7f994c069210 2024-03-11T22:56:35.675 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:35 smithi163 ceph-mon[35584]: pgmap v36: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:35.675 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:35 smithi163 ceph-mon[35584]: Detected new or changed devices on smithi153 2024-03-11T22:56:35.675 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:35 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:35.675 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:35 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:35.675 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:35 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:56:35.675 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:35 smithi163 ceph-mon[35584]: Adjusting osd_memory_target on smithi153 to 12864M 2024-03-11T22:56:35.675 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:35 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:35.676 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:35 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:35.676 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:35 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:56:35.676 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:35 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:35.676 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:35 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:56:35.676 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:35 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:56:35.676 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:35 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:56:35.676 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:35 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:35.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:35 smithi153 ceph-mon[28962]: pgmap v36: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:35.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:35 smithi153 ceph-mon[28962]: Detected new or changed devices on smithi153 2024-03-11T22:56:35.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:35 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:35.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:35 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:35.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:35 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:56:35.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:35 smithi153 ceph-mon[28962]: Adjusting osd_memory_target on smithi153 to 12864M 2024-03-11T22:56:35.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:35 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:35.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:35 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:35.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:35 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:56:35.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:35 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:35.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:35 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:56:35.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:35 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:56:35.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:35 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:56:35.782 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:35 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:36.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:36 smithi153 ceph-mon[28962]: from='client.14252 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:56:36.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:36 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:36.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:36 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:36.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:36 smithi163 ceph-mon[35584]: from='client.14252 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:56:36.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:36 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:36.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:36 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:37.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:37 smithi163 ceph-mon[35584]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:37.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:37 smithi153 ceph-mon[28962]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:39.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:39 smithi153 ceph-mon[28962]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:39.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:39 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/1188439129' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7be06d22-d83a-4b39-ace8-3d800bbd1aee"}]: dispatch 2024-03-11T22:56:39.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:39 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/1188439129' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7be06d22-d83a-4b39-ace8-3d800bbd1aee"}]': finished 2024-03-11T22:56:39.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:39 smithi153 ceph-mon[28962]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T22:56:39.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:39 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:56:39.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:39 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/8376453' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:56:39.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:39 smithi163 ceph-mon[35584]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:39.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:39 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/1188439129' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7be06d22-d83a-4b39-ace8-3d800bbd1aee"}]: dispatch 2024-03-11T22:56:39.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:39 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/1188439129' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7be06d22-d83a-4b39-ace8-3d800bbd1aee"}]': finished 2024-03-11T22:56:39.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:39 smithi163 ceph-mon[35584]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T22:56:39.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:39 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:56:39.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:39 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/8376453' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:56:41.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:41 smithi153 ceph-mon[28962]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:41.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:41 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:41.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:41 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:41.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:41 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:41.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:41 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:56:41.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:41 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:41.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:41 smithi163 ceph-mon[35584]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:41.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:41.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:41.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:41.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:56:41.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:43.506 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:43 smithi153 ceph-mon[28962]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:43.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:43 smithi163 ceph-mon[35584]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:45.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:45 smithi153 ceph-mon[28962]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:45.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:45 smithi163 ceph-mon[35584]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:46.457 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:46 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T22:56:46.457 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:46 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:46.457 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:46 smithi153 ceph-mon[28962]: Deploying daemon osd.1 on smithi153 2024-03-11T22:56:46.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:46 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T22:56:46.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:46 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:46.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:46 smithi163 ceph-mon[35584]: Deploying daemon osd.1 on smithi153 2024-03-11T22:56:47.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:47 smithi153 ceph-mon[28962]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:47.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:47 smithi163 ceph-mon[35584]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:48.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:48 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:56:48.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:48 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:56:49.533 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:49 smithi153 ceph-mon[28962]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:49.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:49 smithi163 ceph-mon[35584]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:51.463 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:51 smithi153 ceph-mon[28962]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:51.463 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:51 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:51.463 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:51 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:51.464 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:51 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:56:51.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:51 smithi163 ceph-mon[35584]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:51.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:51 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:51.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:51 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:51.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:51 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:56:53.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:53 smithi163 ceph-mon[35584]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:53.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:53 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:53.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:53 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:53.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:53 smithi153 ceph-mon[28962]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:53.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:53 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:53.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:53 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:54.486 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 1 on host 'smithi153' 2024-03-11T22:56:54.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:54.479+0000 7f994bfff700 1 -- 172.21.15.153:0/762687155 <== mgr.14204 v2:172.21.15.153:6800/29938766 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f9964115fd0 con 0x7f994c069210 2024-03-11T22:56:54.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:54.481+0000 7f9969ebb700 1 -- 172.21.15.153:0/762687155 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f994c069210 msgr2=0x7f994c06b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:56:54.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:54.481+0000 7f9969ebb700 1 --2- 172.21.15.153:0/762687155 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f994c069210 0x7f994c06b6c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9954004f90 tx=0x7f9954009e30 comp rx=0 tx=0).stop 2024-03-11T22:56:54.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:54.481+0000 7f9969ebb700 1 -- 172.21.15.153:0/762687155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9964071f10 msgr2=0x7f9964114470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:56:54.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:54.481+0000 7f9969ebb700 1 --2- 172.21.15.153:0/762687155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9964071f10 0x7f9964114470 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9958002a50 tx=0x7f99580046b0 comp rx=0 tx=0).stop 2024-03-11T22:56:54.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:54.481+0000 7f9969ebb700 1 -- 172.21.15.153:0/762687155 shutdown_connections 2024-03-11T22:56:54.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:54.482+0000 7f9969ebb700 1 --2- 172.21.15.153:0/762687155 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f994c069210 0x7f994c06b6c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:56:54.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:54.482+0000 7f9969ebb700 1 --2- 172.21.15.153:0/762687155 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f99640708c0 0x7f9964113f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:56:54.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:54.482+0000 7f9969ebb700 1 --2- 172.21.15.153:0/762687155 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9964071f10 0x7f9964114470 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:56:54.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:54.482+0000 7f9969ebb700 1 -- 172.21.15.153:0/762687155 >> 172.21.15.153:0/762687155 conn(0x7f99640f8620 msgr2=0x7f99640fe980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:56:54.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:54.482+0000 7f9969ebb700 1 -- 172.21.15.153:0/762687155 shutdown_connections 2024-03-11T22:56:54.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:56:54.482+0000 7f9969ebb700 1 -- 172.21.15.153:0/762687155 wait complete. 2024-03-11T22:56:55.451 DEBUG:teuthology.orchestra.run.smithi153:osd.1> sudo journalctl -f -n 0 -u ceph-fa913b96-dff9-11ee-95c8-87774f69a715@osd.1.service 2024-03-11T22:56:55.453 INFO:tasks.cephadm:Deploying osd.2 on smithi153 with /dev/vg_nvme/lv_2... 2024-03-11T22:56:55.453 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-11T22:56:55.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:55 smithi163 ceph-mon[35584]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:55.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:55 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:55.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:55 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:55.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:55 smithi163 ceph-mon[35584]: from='osd.1 [v2:172.21.15.153:6810/2718652372,v1:172.21.15.153:6811/2718652372]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T22:56:55.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:55 smithi153 ceph-mon[28962]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:55.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:55 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:55.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:55 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:55.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:55 smithi153 ceph-mon[28962]: from='osd.1 [v2:172.21.15.153:6810/2718652372,v1:172.21.15.153:6811/2718652372]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T22:56:55.861 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:56:56.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:56 smithi153 ceph-mon[28962]: from='osd.1 [v2:172.21.15.153:6810/2718652372,v1:172.21.15.153:6811/2718652372]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T22:56:56.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:56 smithi153 ceph-mon[28962]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T22:56:56.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:56 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:56:56.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:56 smithi153 ceph-mon[28962]: from='osd.1 [v2:172.21.15.153:6810/2718652372,v1:172.21.15.153:6811/2718652372]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-11T22:56:56.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 22:56:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T22:56:56.487+0000 7f9e67ca9700 -1 osd.1 0 waiting for initial osdmap 2024-03-11T22:56:56.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 22:56:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T22:56:56.495+0000 7f9e62aa4700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:56:56.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:56 smithi163 ceph-mon[35584]: from='osd.1 [v2:172.21.15.153:6810/2718652372,v1:172.21.15.153:6811/2718652372]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T22:56:56.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:56 smithi163 ceph-mon[35584]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T22:56:56.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:56 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:56:56.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:56 smithi163 ceph-mon[35584]: from='osd.1 [v2:172.21.15.153:6810/2718652372,v1:172.21.15.153:6811/2718652372]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-11T22:56:57.734 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:57 smithi153 ceph-mon[28962]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:57.734 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:57 smithi153 ceph-mon[28962]: from='osd.1 [v2:172.21.15.153:6810/2718652372,v1:172.21.15.153:6811/2718652372]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-11T22:56:57.734 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:57 smithi153 ceph-mon[28962]: osdmap e13: 2 total, 1 up, 2 in 2024-03-11T22:56:57.734 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:57 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:56:57.734 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:57 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:56:57.889 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:57 smithi163 ceph-mon[35584]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:57.890 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:57 smithi163 ceph-mon[35584]: from='osd.1 [v2:172.21.15.153:6810/2718652372,v1:172.21.15.153:6811/2718652372]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-11T22:56:57.890 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:57 smithi163 ceph-mon[35584]: osdmap e13: 2 total, 1 up, 2 in 2024-03-11T22:56:57.890 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:57 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:56:57.890 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:57 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:56:58.580 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:58 smithi153 ceph-mon[28962]: purged_snaps scrub starts 2024-03-11T22:56:58.580 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:58 smithi153 ceph-mon[28962]: purged_snaps scrub ok 2024-03-11T22:56:58.581 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:58 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:56:58.581 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:58 smithi153 ceph-mon[28962]: osd.1 [v2:172.21.15.153:6810/2718652372,v1:172.21.15.153:6811/2718652372] boot 2024-03-11T22:56:58.581 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:58 smithi153 ceph-mon[28962]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T22:56:58.581 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:58 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:56:58.910 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:58 smithi163 ceph-mon[35584]: purged_snaps scrub starts 2024-03-11T22:56:58.910 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:58 smithi163 ceph-mon[35584]: purged_snaps scrub ok 2024-03-11T22:56:58.910 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:58 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:56:58.910 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:58 smithi163 ceph-mon[35584]: osd.1 [v2:172.21.15.153:6810/2718652372,v1:172.21.15.153:6811/2718652372] boot 2024-03-11T22:56:58.911 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:58 smithi163 ceph-mon[35584]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T22:56:58.911 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:58 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:56:59.053 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-11T22:56:59.053 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-11T22:56:59.053 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-03-11T22:56:59.053 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-03-11T22:56:59.054 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0138189 s, 759 MB/s 2024-03-11T22:56:59.054 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-03-11T22:56:59.602 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:59 smithi153 ceph-mon[28962]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:59.603 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:59 smithi153 ceph-mon[28962]: osdmap e15: 2 total, 2 up, 2 in 2024-03-11T22:56:59.603 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:59 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:59.603 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:56:59 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:59.626 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi153:vg_nvme/lv_2 2024-03-11T22:56:59.990 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:59 smithi163 ceph-mon[35584]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:59.990 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:59 smithi163 ceph-mon[35584]: osdmap e15: 2 total, 2 up, 2 in 2024-03-11T22:56:59.990 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:59 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:56:59.990 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:56:59 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:00.551 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:57:01.759 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:01 smithi153 ceph-mon[28962]: pgmap v54: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-11T22:57:01.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:01 smithi163 ceph-mon[35584]: pgmap v54: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-11T22:57:02.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:02.510+0000 7f4546263700 1 -- 172.21.15.153:0/2186491087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4540066380 msgr2=0x7f45400ffb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:57:02.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:02.510+0000 7f4546263700 1 --2- 172.21.15.153:0/2186491087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4540066380 0x7f45400ffb50 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4538009ce0 tx=0x7f4538004d10 comp rx=0 tx=0).stop 2024-03-11T22:57:02.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:02.510+0000 7f4546263700 1 -- 172.21.15.153:0/2186491087 shutdown_connections 2024-03-11T22:57:02.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:02.510+0000 7f4546263700 1 --2- 172.21.15.153:0/2186491087 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4540100120 0x7f4540102520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:57:02.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:02.510+0000 7f4546263700 1 --2- 172.21.15.153:0/2186491087 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4540066380 0x7f45400ffb50 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:57:02.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:02.510+0000 7f4546263700 1 -- 172.21.15.153:0/2186491087 >> 172.21.15.153:0/2186491087 conn(0x7f45400f74e0 msgr2=0x7f45400f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:57:02.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:02.510+0000 7f4546263700 1 -- 172.21.15.153:0/2186491087 shutdown_connections 2024-03-11T22:57:02.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:02.510+0000 7f4546263700 1 -- 172.21.15.153:0/2186491087 wait complete. 2024-03-11T22:57:02.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:02.510+0000 7f4546263700 1 Processor -- start 2024-03-11T22:57:02.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:02.511+0000 7f4546263700 1 -- start start 2024-03-11T22:57:02.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:02.511+0000 7f4546263700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4540066380 0x7f454010dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:57:02.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:02.511+0000 7f4546263700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4540100120 0x7f454010e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:57:02.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:02.511+0000 7f4546263700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f454010e620 con 0x7f4540066380 2024-03-11T22:57:02.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:02.511+0000 7f4546263700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f454010eec0 con 0x7f4540100120 2024-03-11T22:57:02.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:02.511+0000 7f453ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4540066380 0x7f454010dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:57:02.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:02.511+0000 7f453ffff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4540066380 0x7f454010dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:36772/0 (socket says 172.21.15.153:36772) 2024-03-11T22:57:02.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:02.511+0000 7f453ffff700 1 -- 172.21.15.153:0/2152610693 learned_addr learned my addr 172.21.15.153:0/2152610693 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:57:02.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:02.511+0000 7f453f7fe700 1 --2- 172.21.15.153:0/2152610693 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4540100120 0x7f454010e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:57:02.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:02.511+0000 7f453ffff700 1 -- 172.21.15.153:0/2152610693 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4540100120 msgr2=0x7f454010e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:57:02.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:02.511+0000 7f453ffff700 1 --2- 172.21.15.153:0/2152610693 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4540100120 0x7f454010e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:57:02.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:02.511+0000 7f453ffff700 1 -- 172.21.15.153:0/2152610693 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4538009940 con 0x7f4540066380 2024-03-11T22:57:02.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:02.512+0000 7f453ffff700 1 --2- 172.21.15.153:0/2152610693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4540066380 0x7f454010dae0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4538015040 tx=0x7f4538004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:57:02.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:02.512+0000 7f453cff9700 1 -- 172.21.15.153:0/2152610693 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f453800e040 con 0x7f4540066380 2024-03-11T22:57:02.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:02.512+0000 7f453cff9700 1 -- 172.21.15.153:0/2152610693 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f45380128e0 con 0x7f4540066380 2024-03-11T22:57:02.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:02.512+0000 7f4546263700 1 -- 172.21.15.153:0/2152610693 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f454010f140 con 0x7f4540066380 2024-03-11T22:57:02.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:02.512+0000 7f453cff9700 1 -- 172.21.15.153:0/2152610693 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f453801c760 con 0x7f4540066380 2024-03-11T22:57:02.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:02.512+0000 7f4546263700 1 -- 172.21.15.153:0/2152610693 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f454010f680 con 0x7f4540066380 2024-03-11T22:57:02.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:02.513+0000 7f4546263700 1 -- 172.21.15.153:0/2152610693 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f454004bed0 con 0x7f4540066380 2024-03-11T22:57:02.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:02.513+0000 7f453cff9700 1 -- 172.21.15.153:0/2152610693 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7f4538012410 con 0x7f4540066380 2024-03-11T22:57:02.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:02.514+0000 7f453cff9700 1 --2- 172.21.15.153:0/2152610693 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f4528069210 0x7f452806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:57:02.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:02.514+0000 7f453cff9700 1 -- 172.21.15.153:0/2152610693 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2410+0+0 (secure 0 0 0) 0x7f4538088050 con 0x7f4540066380 2024-03-11T22:57:02.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:02.514+0000 7f453f7fe700 1 --2- 172.21.15.153:0/2152610693 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f4528069210 0x7f452806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:57:02.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:02.514+0000 7f453f7fe700 1 --2- 172.21.15.153:0/2152610693 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f4528069210 0x7f452806b6c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f45400660c0 tx=0x7f453400b040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:57:02.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:02.518+0000 7f453cff9700 1 -- 172.21.15.153:0/2152610693 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f4538058ae0 con 0x7f4540066380 2024-03-11T22:57:02.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:02.721+0000 7f4546263700 1 -- 172.21.15.153:0/2152610693 --> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f454010f2d0 con 0x7f4528069210 2024-03-11T22:57:03.777 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:03 smithi153 ceph-mon[28962]: pgmap v55: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-11T22:57:03.777 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:03 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:57:03.777 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:03 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:57:03.777 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:03 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:03.777 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:03 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:57:03.777 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:03 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:03.778 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:03 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:03.778 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:03 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:57:03.778 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:03 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:57:03.778 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:03 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:03.778 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:03 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:03.778 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:03 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:57:03.778 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:03 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:03.778 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:03 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:57:03.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:03 smithi163 ceph-mon[35584]: pgmap v55: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-11T22:57:03.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:03 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:57:03.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:03 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:57:03.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:03 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:03.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:03 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:57:03.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:03 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:03.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:03 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:03.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:03 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:57:03.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:03 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:57:03.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:03 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:03.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:03 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:03.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:03 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:57:03.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:03 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:03.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:03 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:57:04.714 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:04 smithi153 ceph-mon[28962]: from='client.14270 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:57:04.714 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:04 smithi153 ceph-mon[28962]: Detected new or changed devices on smithi153 2024-03-11T22:57:04.714 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:04 smithi153 ceph-mon[28962]: Adjusting osd_memory_target on smithi153 to 6432M 2024-03-11T22:57:04.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:04 smithi163 ceph-mon[35584]: from='client.14270 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:57:04.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:04 smithi163 ceph-mon[35584]: Detected new or changed devices on smithi153 2024-03-11T22:57:04.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:04 smithi163 ceph-mon[35584]: Adjusting osd_memory_target on smithi153 to 6432M 2024-03-11T22:57:05.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:05 smithi153 ceph-mon[28962]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-11T22:57:05.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:05 smithi163 ceph-mon[35584]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-11T22:57:06.878 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:06 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/1040610440' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4863162-3a3d-45ae-9bc0-c8b069ff063a"}]: dispatch 2024-03-11T22:57:06.878 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:06 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/1040610440' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b4863162-3a3d-45ae-9bc0-c8b069ff063a"}]': finished 2024-03-11T22:57:06.878 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:06 smithi153 ceph-mon[28962]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T22:57:06.878 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:06 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:57:06.878 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:06 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/1710006071' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:57:06.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:06 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/1040610440' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4863162-3a3d-45ae-9bc0-c8b069ff063a"}]: dispatch 2024-03-11T22:57:06.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:06 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/1040610440' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b4863162-3a3d-45ae-9bc0-c8b069ff063a"}]': finished 2024-03-11T22:57:06.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:06 smithi163 ceph-mon[35584]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T22:57:06.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:06 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:57:06.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:06 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/1710006071' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:57:07.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:07 smithi153 ceph-mon[28962]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-11T22:57:07.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:07 smithi163 ceph-mon[35584]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-11T22:57:09.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:09 smithi153 ceph-mon[28962]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-11T22:57:09.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:09 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:09.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:09 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:09.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:09 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:09.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:09 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:57:09.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:09 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:09.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:09 smithi163 ceph-mon[35584]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-11T22:57:09.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:09 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:09.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:09 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:09.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:09 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:09.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:09 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:57:09.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:09 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:11.897 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:11 smithi153 ceph-mon[28962]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-11T22:57:11.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:11 smithi163 ceph-mon[35584]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-11T22:57:13.934 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:13 smithi153 ceph-mon[28962]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-11T22:57:13.934 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:13 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T22:57:13.934 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:13 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:13.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:13 smithi163 ceph-mon[35584]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-11T22:57:13.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:13 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T22:57:13.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:13 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:14.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:14 smithi153 ceph-mon[28962]: Deploying daemon osd.2 on smithi153 2024-03-11T22:57:14.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:14 smithi163 ceph-mon[35584]: Deploying daemon osd.2 on smithi153 2024-03-11T22:57:15.731 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:15 smithi153 ceph-mon[28962]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-11T22:57:15.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:15 smithi163 ceph-mon[35584]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-11T22:57:16.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:16 smithi153 ceph-mon[28962]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-11T22:57:16.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:16 smithi163 ceph-mon[35584]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-11T22:57:19.238 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:18 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:19.238 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:18 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:19.238 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:18 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:57:19.238 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:18 smithi163 ceph-mon[35584]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-11T22:57:19.238 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:18 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:57:19.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:18 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:19.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:18 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:19.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:18 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:57:19.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:18 smithi153 ceph-mon[28962]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-11T22:57:19.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:18 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:57:21.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:21 smithi153 ceph-mon[28962]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-11T22:57:21.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:21 smithi163 ceph-mon[35584]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-11T22:57:21.927 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 2 on host 'smithi153' 2024-03-11T22:57:21.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:21.922+0000 7f453cff9700 1 -- 172.21.15.153:0/2152610693 <== mgr.14204 v2:172.21.15.153:6800/29938766 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f454010f2d0 con 0x7f4528069210 2024-03-11T22:57:21.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:21.924+0000 7f4546263700 1 -- 172.21.15.153:0/2152610693 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f4528069210 msgr2=0x7f452806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:57:21.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:21.924+0000 7f4546263700 1 --2- 172.21.15.153:0/2152610693 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f4528069210 0x7f452806b6c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f45400660c0 tx=0x7f453400b040 comp rx=0 tx=0).stop 2024-03-11T22:57:21.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:21.925+0000 7f4546263700 1 -- 172.21.15.153:0/2152610693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4540066380 msgr2=0x7f454010dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:57:21.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:21.925+0000 7f4546263700 1 --2- 172.21.15.153:0/2152610693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4540066380 0x7f454010dae0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4538015040 tx=0x7f4538004d10 comp rx=0 tx=0).stop 2024-03-11T22:57:21.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:21.925+0000 7f4546263700 1 -- 172.21.15.153:0/2152610693 shutdown_connections 2024-03-11T22:57:21.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:21.925+0000 7f4546263700 1 --2- 172.21.15.153:0/2152610693 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4540066380 0x7f454010dae0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:57:21.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:21.925+0000 7f4546263700 1 --2- 172.21.15.153:0/2152610693 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f4528069210 0x7f452806b6c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:57:21.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:21.925+0000 7f4546263700 1 --2- 172.21.15.153:0/2152610693 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4540100120 0x7f454010e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:57:21.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:21.925+0000 7f4546263700 1 -- 172.21.15.153:0/2152610693 >> 172.21.15.153:0/2152610693 conn(0x7f45400f74e0 msgr2=0x7f45400f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:57:21.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:21.925+0000 7f4546263700 1 -- 172.21.15.153:0/2152610693 shutdown_connections 2024-03-11T22:57:21.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:57:21.925+0000 7f4546263700 1 -- 172.21.15.153:0/2152610693 wait complete. 2024-03-11T22:57:22.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:22 smithi153 ceph-mon[28962]: from='osd.2 [v2:172.21.15.153:6818/4016153774,v1:172.21.15.153:6819/4016153774]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T22:57:22.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:22 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:22.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:22 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:22.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:22 smithi163 ceph-mon[35584]: from='osd.2 [v2:172.21.15.153:6818/4016153774,v1:172.21.15.153:6819/4016153774]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T22:57:22.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:22 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:22.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:22 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:22.930 DEBUG:teuthology.orchestra.run.smithi153:osd.2> sudo journalctl -f -n 0 -u ceph-fa913b96-dff9-11ee-95c8-87774f69a715@osd.2.service 2024-03-11T22:57:22.932 INFO:tasks.cephadm:Deploying osd.3 on smithi163 with /dev/vg_nvme/lv_4... 2024-03-11T22:57:22.932 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-11T22:57:23.193 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi163/config 2024-03-11T22:57:23.538 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:23 smithi163 ceph-mon[35584]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-11T22:57:23.538 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:23 smithi163 ceph-mon[35584]: from='osd.2 [v2:172.21.15.153:6818/4016153774,v1:172.21.15.153:6819/4016153774]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T22:57:23.538 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:23 smithi163 ceph-mon[35584]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T22:57:23.538 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:23 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:57:23.538 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:23 smithi163 ceph-mon[35584]: from='osd.2 [v2:172.21.15.153:6818/4016153774,v1:172.21.15.153:6819/4016153774]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-11T22:57:23.668 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:23 smithi153 ceph-mon[28962]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-11T22:57:23.669 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:23 smithi153 ceph-mon[28962]: from='osd.2 [v2:172.21.15.153:6818/4016153774,v1:172.21.15.153:6819/4016153774]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T22:57:23.669 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:23 smithi153 ceph-mon[28962]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T22:57:23.669 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:23 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:57:23.669 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:23 smithi153 ceph-mon[28962]: from='osd.2 [v2:172.21.15.153:6818/4016153774,v1:172.21.15.153:6819/4016153774]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-11T22:57:23.669 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 22:57:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T22:57:23.286+0000 7fb4cbee4700 -1 osd.2 0 waiting for initial osdmap 2024-03-11T22:57:23.669 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 22:57:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T22:57:23.290+0000 7fb4c84e2700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:57:24.426 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:24 smithi163 ceph-mon[35584]: from='osd.2 [v2:172.21.15.153:6818/4016153774,v1:172.21.15.153:6819/4016153774]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-11T22:57:24.426 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:24 smithi163 ceph-mon[35584]: osdmap e18: 3 total, 2 up, 3 in 2024-03-11T22:57:24.426 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:24 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:57:24.426 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:24 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:57:24.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:24 smithi153 ceph-mon[28962]: from='osd.2 [v2:172.21.15.153:6818/4016153774,v1:172.21.15.153:6819/4016153774]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-11T22:57:24.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:24 smithi153 ceph-mon[28962]: osdmap e18: 3 total, 2 up, 3 in 2024-03-11T22:57:24.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:24 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:57:24.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:24 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:57:24.968 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-11T22:57:24.968 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-11T22:57:24.968 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10+0 records in 2024-03-11T22:57:24.968 INFO:teuthology.orchestra.run.smithi163.stderr:10+0 records out 2024-03-11T22:57:24.968 INFO:teuthology.orchestra.run.smithi163.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113537 s, 924 MB/s 2024-03-11T22:57:24.968 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping successful for: 2024-03-11T22:57:25.524 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:25 smithi153 ceph-mon[28962]: purged_snaps scrub starts 2024-03-11T22:57:25.524 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:25 smithi153 ceph-mon[28962]: purged_snaps scrub ok 2024-03-11T22:57:25.524 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:25 smithi153 ceph-mon[28962]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-11T22:57:25.524 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:25 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:57:25.524 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:25 smithi153 ceph-mon[28962]: osd.2 [v2:172.21.15.153:6818/4016153774,v1:172.21.15.153:6819/4016153774] boot 2024-03-11T22:57:25.524 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:25 smithi153 ceph-mon[28962]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T22:57:25.524 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:25 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:57:25.525 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:25 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:25.525 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:25 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:25.552 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:25 smithi163 ceph-mon[35584]: purged_snaps scrub starts 2024-03-11T22:57:25.552 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:25 smithi163 ceph-mon[35584]: purged_snaps scrub ok 2024-03-11T22:57:25.552 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:25 smithi163 ceph-mon[35584]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-11T22:57:25.552 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:25 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:57:25.552 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:25 smithi163 ceph-mon[35584]: osd.2 [v2:172.21.15.153:6818/4016153774,v1:172.21.15.153:6819/4016153774] boot 2024-03-11T22:57:25.553 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:25 smithi163 ceph-mon[35584]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T22:57:25.553 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:25 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:57:25.553 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:25 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:25.553 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:25 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:25.577 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi163:vg_nvme/lv_4 2024-03-11T22:57:25.843 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi163/config 2024-03-11T22:57:26.470 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:26 smithi153 ceph-mon[28962]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T22:57:26.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:26 smithi163 ceph-mon[35584]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T22:57:27.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:27.356+0000 7f8edc916700 1 -- 172.21.15.163:0/796510787 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8ed80fe0e0 msgr2=0x7f8ed80fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:57:27.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:27.356+0000 7f8edc916700 1 --2- 172.21.15.163:0/796510787 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8ed80fe0e0 0x7f8ed80fe550 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc004970 tx=0x7f8ecc009e30 comp rx=0 tx=0).stop 2024-03-11T22:57:27.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:27.356+0000 7f8edc916700 1 -- 172.21.15.163:0/796510787 shutdown_connections 2024-03-11T22:57:27.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:27.356+0000 7f8edc916700 1 --2- 172.21.15.163:0/796510787 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8ed80fe0e0 0x7f8ed80fe550 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:57:27.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:27.356+0000 7f8edc916700 1 --2- 172.21.15.163:0/796510787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ed80fcef0 0x7f8ed80fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:57:27.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:27.356+0000 7f8edc916700 1 -- 172.21.15.163:0/796510787 >> 172.21.15.163:0/796510787 conn(0x7f8ed80f8620 msgr2=0x7f8ed80faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:57:27.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:27.356+0000 7f8edc916700 1 -- 172.21.15.163:0/796510787 shutdown_connections 2024-03-11T22:57:27.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:27.356+0000 7f8edc916700 1 -- 172.21.15.163:0/796510787 wait complete. 2024-03-11T22:57:27.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:27.357+0000 7f8edc916700 1 Processor -- start 2024-03-11T22:57:27.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:27.357+0000 7f8edc916700 1 -- start start 2024-03-11T22:57:27.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:27.357+0000 7f8edc916700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8ed80fcef0 0x7f8ed8071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:57:27.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:27.357+0000 7f8edc916700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ed80fe0e0 0x7f8ed80702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:57:27.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:27.357+0000 7f8edc916700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed80708e0 con 0x7f8ed80fe0e0 2024-03-11T22:57:27.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:27.357+0000 7f8edc916700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed8070a50 con 0x7f8ed80fcef0 2024-03-11T22:57:27.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:27.357+0000 7f8ed659c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8ed80fcef0 0x7f8ed8071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:57:27.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:27.357+0000 7f8ed659c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8ed80fcef0 0x7f8ed8071e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:57796/0 (socket says 172.21.15.163:57796) 2024-03-11T22:57:27.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:27.357+0000 7f8ed659c700 1 -- 172.21.15.163:0/2238765842 learned_addr learned my addr 172.21.15.163:0/2238765842 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-03-11T22:57:27.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:27.357+0000 7f8ed5d9b700 1 --2- 172.21.15.163:0/2238765842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ed80fe0e0 0x7f8ed80702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:57:27.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:27.357+0000 7f8ed659c700 1 -- 172.21.15.163:0/2238765842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ed80fe0e0 msgr2=0x7f8ed80702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:57:27.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:27.357+0000 7f8ed659c700 1 --2- 172.21.15.163:0/2238765842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ed80fe0e0 0x7f8ed80702e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:57:27.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:27.357+0000 7f8ed659c700 1 -- 172.21.15.163:0/2238765842 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ecc005040 con 0x7f8ed80fcef0 2024-03-11T22:57:27.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:27.357+0000 7f8ed659c700 1 --2- 172.21.15.163:0/2238765842 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8ed80fcef0 0x7f8ed8071e60 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f8ec8002a80 tx=0x7f8ec8004600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:57:27.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:27.357+0000 7f8ec6ffd700 1 -- 172.21.15.163:0/2238765842 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ec8007cc0 con 0x7f8ed80fcef0 2024-03-11T22:57:27.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:27.357+0000 7f8ec6ffd700 1 -- 172.21.15.163:0/2238765842 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ec800f040 con 0x7f8ed80fcef0 2024-03-11T22:57:27.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:27.357+0000 7f8edc916700 1 -- 172.21.15.163:0/2238765842 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ed8070d30 con 0x7f8ed80fcef0 2024-03-11T22:57:27.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:27.358+0000 7f8ec6ffd700 1 -- 172.21.15.163:0/2238765842 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8ec800da30 con 0x7f8ed80fcef0 2024-03-11T22:57:27.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:27.358+0000 7f8edc916700 1 -- 172.21.15.163:0/2238765842 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ed8115940 con 0x7f8ed80fcef0 2024-03-11T22:57:27.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:27.358+0000 7f8edc916700 1 -- 172.21.15.163:0/2238765842 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ed804bed0 con 0x7f8ed80fcef0 2024-03-11T22:57:27.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:27.359+0000 7f8ec6ffd700 1 -- 172.21.15.163:0/2238765842 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7f8ec800db90 con 0x7f8ed80fcef0 2024-03-11T22:57:27.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:27.359+0000 7f8ec6ffd700 1 --2- 172.21.15.163:0/2238765842 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f8ec0069210 0x7f8ec006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:57:27.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:27.359+0000 7f8ec6ffd700 1 -- 172.21.15.163:0/2238765842 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3217+0+0 (secure 0 0 0) 0x7f8ec8086470 con 0x7f8ed80fcef0 2024-03-11T22:57:27.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:27.360+0000 7f8ed5d9b700 1 --2- 172.21.15.163:0/2238765842 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f8ec0069210 0x7f8ec006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:57:27.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:27.360+0000 7f8ed5d9b700 1 --2- 172.21.15.163:0/2238765842 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f8ec0069210 0x7f8ec006b6c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc004940 tx=0x7f8ecc009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:57:27.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:27.364+0000 7f8ec6ffd700 1 -- 172.21.15.163:0/2238765842 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f8ec8056bd0 con 0x7f8ed80fcef0 2024-03-11T22:57:27.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:27 smithi163 ceph-mon[35584]: pgmap v72: 0 pgs: ; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-03-11T22:57:27.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:27 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-11T22:57:27.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:27 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-11T22:57:27.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:27 smithi163 ceph-mon[35584]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T22:57:27.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:27 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:57:27.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:27 smithi153 ceph-mon[28962]: pgmap v72: 0 pgs: ; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-03-11T22:57:27.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:27 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-11T22:57:27.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:27 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-11T22:57:27.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:27 smithi153 ceph-mon[28962]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T22:57:27.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:27 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:57:27.566 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:27.561+0000 7f8edc916700 1 -- 172.21.15.163:0/2238765842 --> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ed8102ac0 con 0x7f8ec0069210 2024-03-11T22:57:28.242 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 22:57:27 smithi153 sudo[65938]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T22:57:28.242 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 22:57:27 smithi153 sudo[65938]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:57:28.242 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 22:57:27 smithi153 sudo[65938]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:57:28.243 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 22:57:27 smithi153 sudo[65938]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:57:28.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 22:57:28 smithi153 sudo[66027]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T22:57:28.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 22:57:28 smithi153 sudo[66027]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:57:28.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 22:57:28 smithi153 sudo[66027]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:57:28.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 22:57:28 smithi153 sudo[66027]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:57:28.526 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:28 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T22:57:28.527 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:28 smithi153 ceph-mon[28962]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T22:57:28.527 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:28 smithi153 ceph-mon[28962]: from='client.24127 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:57:28.527 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:28 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:57:28.527 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:28 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:57:28.527 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:28 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:28.527 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:28 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:28.527 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:28 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:28.527 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:28 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:57:28.527 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:28 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:57:28.527 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:28 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:57:28.527 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:28 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:28.527 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:28 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:28.527 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:28 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:57:28.527 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:28 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:28.528 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:28 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:57:28.528 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:28 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:28.668 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:28 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T22:57:28.668 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:28 smithi163 ceph-mon[35584]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T22:57:28.668 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:28 smithi163 ceph-mon[35584]: from='client.24127 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:57:28.668 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:28 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:57:28.668 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:28 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:57:28.668 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:28 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:28.669 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:28 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:28.669 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:28 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:28.669 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:28 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:57:28.669 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:28 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:57:28.669 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:28 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:57:28.669 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:28 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:28.669 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:28 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:28.669 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:28 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:57:28.669 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:28 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:28.669 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:28 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:57:28.669 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:28 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:28.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 22:57:28 smithi153 sudo[66116]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T22:57:28.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 22:57:28 smithi153 sudo[66116]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:57:28.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 22:57:28 smithi153 sudo[66116]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:57:28.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 22:57:28 smithi153 sudo[66116]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:57:29.039 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 22:57:28 smithi153 sudo[66138]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T22:57:29.040 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 22:57:28 smithi153 sudo[66138]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:57:29.040 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 22:57:28 smithi153 sudo[66138]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:57:29.040 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 22:57:28 smithi153 sudo[66138]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:57:29.040 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 22:57:29 smithi153 sudo[66141]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T22:57:29.040 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 22:57:29 smithi153 sudo[66141]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:57:29.040 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 22:57:29 smithi153 sudo[66141]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:57:29.315 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 22:57:29 smithi153 sudo[66141]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:57:29.566 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:29 smithi153 sudo[66250]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T22:57:29.566 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 22:57:29 smithi153 sudo[66227]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T22:57:29.566 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 22:57:29 smithi153 sudo[66227]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:57:29.566 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 22:57:29 smithi153 sudo[66227]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:57:29.566 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 22:57:29 smithi153 sudo[66227]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:57:29.827 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:29 smithi153 sudo[66250]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:57:29.827 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:29 smithi153 sudo[66250]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:57:29.828 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:29 smithi153 sudo[66250]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:57:30.091 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:29 smithi163 ceph-mon[35584]: Detected new or changed devices on smithi153 2024-03-11T22:57:30.091 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:29 smithi163 ceph-mon[35584]: Adjusting osd_memory_target on smithi153 to 4288M 2024-03-11T22:57:30.091 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:29 smithi163 ceph-mon[35584]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-03-11T22:57:30.091 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:29 smithi163 ceph-mon[35584]: osdmap e23: 3 total, 3 up, 3 in 2024-03-11T22:57:30.091 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:29 smithi163 ceph-mon[35584]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:57:30.091 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:30 smithi163 sudo[38926]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T22:57:30.091 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:30 smithi163 sudo[38926]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:57:30.091 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:30 smithi163 sudo[38926]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:57:30.091 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:30 smithi163 sudo[38926]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:57:30.177 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:29 smithi153 sudo[66255]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T22:57:30.177 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:29 smithi153 sudo[66255]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:57:30.177 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:29 smithi153 sudo[66255]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:57:30.177 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:29 smithi153 sudo[66255]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:57:30.177 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:29 smithi153 ceph-mon[28962]: Detected new or changed devices on smithi153 2024-03-11T22:57:30.177 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:29 smithi153 ceph-mon[28962]: Adjusting osd_memory_target on smithi153 to 4288M 2024-03-11T22:57:30.177 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:29 smithi153 ceph-mon[28962]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-03-11T22:57:30.177 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:29 smithi153 ceph-mon[28962]: osdmap e23: 3 total, 3 up, 3 in 2024-03-11T22:57:30.177 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:29 smithi153 ceph-mon[28962]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:57:30.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:30.329+0000 7f8ec6ffd700 1 -- 172.21.15.163:0/2238765842 <== mon.1 v2:172.21.15.163:3300/0 7 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f8ec8053510 con 0x7f8ed80fcef0 2024-03-11T22:57:30.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:30 smithi163 sudo[38929]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T22:57:30.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:30 smithi163 sudo[38929]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:57:30.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:30 smithi163 sudo[38929]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:57:30.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:30 smithi163 sudo[38929]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:57:30.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:30 smithi163 ceph-mon[35584]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:57:30.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:30 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-03-11T22:57:30.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:30 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mon metadata", "id": "smithi163"}]: dispatch 2024-03-11T22:57:30.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:30 smithi163 ceph-mon[35584]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:57:30.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:30 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-03-11T22:57:30.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:30 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mon metadata", "id": "smithi163"}]: dispatch 2024-03-11T22:57:30.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:30 smithi163 ceph-mon[35584]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 872 MiB used, 267 GiB / 268 GiB avail 2024-03-11T22:57:30.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:30 smithi163 ceph-mon[35584]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:57:30.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:30 smithi163 ceph-mon[35584]: from='client.? 172.21.15.163:0/2406910070' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "92a62b43-e245-4d71-b4ca-89a7d518af50"}]: dispatch 2024-03-11T22:57:30.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:30 smithi163 ceph-mon[35584]: mgrmap e19: smithi153.nyocep(active, since 2m), standbys: smithi163.ivtqrh 2024-03-11T22:57:30.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:30 smithi163 ceph-mon[35584]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "92a62b43-e245-4d71-b4ca-89a7d518af50"}]: dispatch 2024-03-11T22:57:30.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:30 smithi163 ceph-mon[35584]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "92a62b43-e245-4d71-b4ca-89a7d518af50"}]': finished 2024-03-11T22:57:30.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:30 smithi163 ceph-mon[35584]: osdmap e24: 4 total, 3 up, 4 in 2024-03-11T22:57:30.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:30 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:57:30.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:30 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:31.092 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:30 smithi153 ceph-mon[28962]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:57:31.092 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:30 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-03-11T22:57:31.092 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:30 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mon metadata", "id": "smithi163"}]: dispatch 2024-03-11T22:57:31.092 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:30 smithi153 ceph-mon[28962]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:57:31.093 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:30 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-03-11T22:57:31.093 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:30 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mon metadata", "id": "smithi163"}]: dispatch 2024-03-11T22:57:31.093 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:30 smithi153 ceph-mon[28962]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 872 MiB used, 267 GiB / 268 GiB avail 2024-03-11T22:57:31.093 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:30 smithi153 ceph-mon[28962]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:57:31.093 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:30 smithi153 ceph-mon[28962]: from='client.? 172.21.15.163:0/2406910070' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "92a62b43-e245-4d71-b4ca-89a7d518af50"}]: dispatch 2024-03-11T22:57:31.093 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:30 smithi153 ceph-mon[28962]: mgrmap e19: smithi153.nyocep(active, since 2m), standbys: smithi163.ivtqrh 2024-03-11T22:57:31.093 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:30 smithi153 ceph-mon[28962]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "92a62b43-e245-4d71-b4ca-89a7d518af50"}]: dispatch 2024-03-11T22:57:31.093 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:30 smithi153 ceph-mon[28962]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "92a62b43-e245-4d71-b4ca-89a7d518af50"}]': finished 2024-03-11T22:57:31.093 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:30 smithi153 ceph-mon[28962]: osdmap e24: 4 total, 3 up, 4 in 2024-03-11T22:57:31.093 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:30 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:57:31.093 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:30 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:32.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:31 smithi153 ceph-mon[28962]: from='client.? 172.21.15.163:0/2083880802' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:57:32.238 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:31 smithi163 ceph-mon[35584]: from='client.? 172.21.15.163:0/2083880802' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:57:33.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:33 smithi163 ceph-mon[35584]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-03-11T22:57:33.489 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:33 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:33.489 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:33 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:33.489 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:33 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:33.489 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:33 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:57:33.489 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:33 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:33.489 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:33 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:57:33.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:33 smithi153 ceph-mon[28962]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-03-11T22:57:33.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:33 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:33.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:33 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:33.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:33 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:33.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:33 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:57:33.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:33 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:33.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:33 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:57:35.504 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:35 smithi163 ceph-mon[35584]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-03-11T22:57:35.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:35 smithi153 ceph-mon[28962]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-03-11T22:57:37.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:37 smithi163 ceph-mon[35584]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-11T22:57:37.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:37 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T22:57:37.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:37 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:37.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:37 smithi153 ceph-mon[28962]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-11T22:57:37.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:37 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T22:57:37.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:37 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:38.634 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:38 smithi163 ceph-mon[35584]: Deploying daemon osd.3 on smithi163 2024-03-11T22:57:38.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:38 smithi153 ceph-mon[28962]: Deploying daemon osd.3 on smithi163 2024-03-11T22:57:39.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:39 smithi163 ceph-mon[35584]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-11T22:57:39.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:39 smithi153 ceph-mon[28962]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-11T22:57:41.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:41 smithi163 ceph-mon[35584]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-11T22:57:41.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:41 smithi153 ceph-mon[28962]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-11T22:57:43.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:43 smithi163 ceph-mon[35584]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 67 KiB/s wr, 4 op/s 2024-03-11T22:57:43.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:43 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:43.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:43 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:43.677 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:43 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:57:43.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:43 smithi153 ceph-mon[28962]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 67 KiB/s wr, 4 op/s 2024-03-11T22:57:43.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:43 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:43.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:43 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:43.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:43 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:57:45.329 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:45.325+0000 7f8ec6ffd700 1 -- 172.21.15.163:0/2238765842 <== mgr.14204 v2:172.21.15.153:6800/29938766 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8ed8102ac0 con 0x7f8ec0069210 2024-03-11T22:57:45.330 INFO:teuthology.orchestra.run.smithi163.stdout:Created osd(s) 3 on host 'smithi163' 2024-03-11T22:57:45.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:45.327+0000 7f8edc916700 1 -- 172.21.15.163:0/2238765842 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f8ec0069210 msgr2=0x7f8ec006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:57:45.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:45.327+0000 7f8edc916700 1 --2- 172.21.15.163:0/2238765842 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f8ec0069210 0x7f8ec006b6c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc004940 tx=0x7f8ecc009e30 comp rx=0 tx=0).stop 2024-03-11T22:57:45.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:45.327+0000 7f8edc916700 1 -- 172.21.15.163:0/2238765842 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8ed80fcef0 msgr2=0x7f8ed8071e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:57:45.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:45.327+0000 7f8edc916700 1 --2- 172.21.15.163:0/2238765842 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8ed80fcef0 0x7f8ed8071e60 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f8ec8002a80 tx=0x7f8ec8004600 comp rx=0 tx=0).stop 2024-03-11T22:57:45.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:45.327+0000 7f8edc916700 1 -- 172.21.15.163:0/2238765842 shutdown_connections 2024-03-11T22:57:45.331 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:45.327+0000 7f8edc916700 1 --2- 172.21.15.163:0/2238765842 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f8ec0069210 0x7f8ec006b6c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:57:45.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:45.327+0000 7f8edc916700 1 --2- 172.21.15.163:0/2238765842 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8ed80fcef0 0x7f8ed8071e60 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:57:45.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:45.327+0000 7f8edc916700 1 --2- 172.21.15.163:0/2238765842 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8ed80fe0e0 0x7f8ed80702e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:57:45.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:45.327+0000 7f8edc916700 1 -- 172.21.15.163:0/2238765842 >> 172.21.15.163:0/2238765842 conn(0x7f8ed80f8620 msgr2=0x7f8ed81013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:57:45.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:45.327+0000 7f8edc916700 1 -- 172.21.15.163:0/2238765842 shutdown_connections 2024-03-11T22:57:45.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:45.327+0000 7f8edc916700 1 -- 172.21.15.163:0/2238765842 wait complete. 2024-03-11T22:57:45.699 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:45 smithi163 ceph-mon[35584]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-11T22:57:45.699 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:45 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:45.699 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:45 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:45.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:45 smithi153 ceph-mon[28962]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-11T22:57:45.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:45 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:45.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:45 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:45.956 DEBUG:teuthology.orchestra.run.smithi163:osd.3> sudo journalctl -f -n 0 -u ceph-fa913b96-dff9-11ee-95c8-87774f69a715@osd.3.service 2024-03-11T22:57:45.958 INFO:tasks.cephadm:Deploying osd.4 on smithi163 with /dev/vg_nvme/lv_3... 2024-03-11T22:57:45.959 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-11T22:57:46.605 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi163/config 2024-03-11T22:57:46.712 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:46 smithi163 ceph-mon[35584]: from='osd.3 [v2:172.21.15.163:6800/3768791983,v1:172.21.15.163:6801/3768791983]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T22:57:46.713 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:46 smithi163 ceph-mon[35584]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T22:57:46.713 INFO:journalctl@ceph.osd.3.smithi163.stdout:Mar 11 22:57:46 smithi163 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-3[41013]: 2024-03-11T22:57:46.305+0000 7f9d4486e540 -1 osd.3 0 log_to_monitors true 2024-03-11T22:57:46.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:46 smithi153 ceph-mon[28962]: from='osd.3 [v2:172.21.15.163:6800/3768791983,v1:172.21.15.163:6801/3768791983]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T22:57:46.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:46 smithi153 ceph-mon[28962]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T22:57:47.684 INFO:journalctl@ceph.osd.3.smithi163.stdout:Mar 11 22:57:47 smithi163 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-3[41013]: 2024-03-11T22:57:47.429+0000 7f9d3b040700 -1 osd.3 0 waiting for initial osdmap 2024-03-11T22:57:47.685 INFO:journalctl@ceph.osd.3.smithi163.stdout:Mar 11 22:57:47 smithi163 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-3[41013]: 2024-03-11T22:57:47.436+0000 7f9d37e3f700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:57:47.685 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:47 smithi163 ceph-mon[35584]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-11T22:57:47.685 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:47 smithi163 ceph-mon[35584]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T22:57:47.685 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:47 smithi163 ceph-mon[35584]: osdmap e25: 4 total, 3 up, 4 in 2024-03-11T22:57:47.685 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:47 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:57:47.685 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:47 smithi163 ceph-mon[35584]: from='osd.3 [v2:172.21.15.163:6800/3768791983,v1:172.21.15.163:6801/3768791983]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-03-11T22:57:47.685 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:47 smithi163 ceph-mon[35584]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-03-11T22:57:47.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:47 smithi153 ceph-mon[28962]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-11T22:57:47.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:47 smithi153 ceph-mon[28962]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T22:57:47.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:47 smithi153 ceph-mon[28962]: osdmap e25: 4 total, 3 up, 4 in 2024-03-11T22:57:47.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:47 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:57:47.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:47 smithi153 ceph-mon[28962]: from='osd.3 [v2:172.21.15.163:6800/3768791983,v1:172.21.15.163:6801/3768791983]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-03-11T22:57:47.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:47 smithi153 ceph-mon[28962]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-03-11T22:57:48.574 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-11T22:57:48.574 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-11T22:57:48.574 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10+0 records in 2024-03-11T22:57:48.574 INFO:teuthology.orchestra.run.smithi163.stderr:10+0 records out 2024-03-11T22:57:48.575 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122295 s, 857 MB/s 2024-03-11T22:57:48.575 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping successful for: 2024-03-11T22:57:48.699 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:48 smithi163 ceph-mon[35584]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-03-11T22:57:48.699 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:48 smithi163 ceph-mon[35584]: osdmap e26: 4 total, 3 up, 4 in 2024-03-11T22:57:48.699 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:48 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:57:48.700 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:48 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:57:48.700 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:48 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:48.700 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:48 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:48.700 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:48 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:57:48.700 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:48 smithi163 ceph-mon[35584]: osd.3 [v2:172.21.15.163:6800/3768791983,v1:172.21.15.163:6801/3768791983] boot 2024-03-11T22:57:48.700 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:48 smithi163 ceph-mon[35584]: osdmap e27: 4 total, 4 up, 4 in 2024-03-11T22:57:48.700 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:48 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:57:48.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:48 smithi153 ceph-mon[28962]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-03-11T22:57:48.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:48 smithi153 ceph-mon[28962]: osdmap e26: 4 total, 3 up, 4 in 2024-03-11T22:57:48.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:48 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:57:48.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:48 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:57:48.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:48 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:48.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:48 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:48.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:48 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:57:48.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:48 smithi153 ceph-mon[28962]: osd.3 [v2:172.21.15.163:6800/3768791983,v1:172.21.15.163:6801/3768791983] boot 2024-03-11T22:57:48.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:48 smithi153 ceph-mon[28962]: osdmap e27: 4 total, 4 up, 4 in 2024-03-11T22:57:48.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:48 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T22:57:49.542 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:49 smithi163 ceph-mon[35584]: purged_snaps scrub starts 2024-03-11T22:57:49.542 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:49 smithi163 ceph-mon[35584]: purged_snaps scrub ok 2024-03-11T22:57:49.542 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:49 smithi163 ceph-mon[35584]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-11T22:57:49.542 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:49 smithi163 ceph-mon[35584]: osdmap e28: 4 total, 4 up, 4 in 2024-03-11T22:57:49.543 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi163:vg_nvme/lv_3 2024-03-11T22:57:49.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:49 smithi153 ceph-mon[28962]: purged_snaps scrub starts 2024-03-11T22:57:49.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:49 smithi153 ceph-mon[28962]: purged_snaps scrub ok 2024-03-11T22:57:49.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:49 smithi153 ceph-mon[28962]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-11T22:57:49.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:49 smithi153 ceph-mon[28962]: osdmap e28: 4 total, 4 up, 4 in 2024-03-11T22:57:50.030 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi163/config 2024-03-11T22:57:51.559 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:51 smithi163 ceph-mon[35584]: pgmap v92: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-11T22:57:51.560 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:51 smithi163 ceph-mon[35584]: osdmap e29: 4 total, 4 up, 4 in 2024-03-11T22:57:51.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:51 smithi153 ceph-mon[28962]: pgmap v92: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-11T22:57:51.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:51 smithi153 ceph-mon[28962]: osdmap e29: 4 total, 4 up, 4 in 2024-03-11T22:57:52.615 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:52 smithi163 ceph-mon[35584]: osdmap e30: 4 total, 4 up, 4 in 2024-03-11T22:57:52.615 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:52 smithi163 ceph-mon[35584]: Detected new or changed devices on smithi163 2024-03-11T22:57:52.615 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:52 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:52.615 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:52 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:52.616 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:52 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:57:52.616 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:52 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:52.616 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:52 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:52.616 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:52 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:57:52.616 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:52 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:52.616 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:52 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:57:52.616 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:52 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:52.616 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:52 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:57:52.617 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:52 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:52.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:52.629+0000 7fce5c59f700 1 -- 172.21.15.163:0/3668797216 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce540fcef0 msgr2=0x7fce540fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:57:52.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:52.629+0000 7fce5c59f700 1 --2- 172.21.15.163:0/3668797216 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce540fcef0 0x7fce540fd320 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fce50009d10 tx=0x7fce50004d10 comp rx=0 tx=0).stop 2024-03-11T22:57:52.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:52.629+0000 7fce5c59f700 1 -- 172.21.15.163:0/3668797216 shutdown_connections 2024-03-11T22:57:52.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:52.629+0000 7fce5c59f700 1 --2- 172.21.15.163:0/3668797216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce540fe0e0 0x7fce540fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:57:52.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:52.629+0000 7fce5c59f700 1 --2- 172.21.15.163:0/3668797216 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce540fcef0 0x7fce540fd320 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:57:52.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:52.629+0000 7fce5c59f700 1 -- 172.21.15.163:0/3668797216 >> 172.21.15.163:0/3668797216 conn(0x7fce540f8620 msgr2=0x7fce540faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:57:52.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:52.629+0000 7fce5c59f700 1 -- 172.21.15.163:0/3668797216 shutdown_connections 2024-03-11T22:57:52.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:52.629+0000 7fce5c59f700 1 -- 172.21.15.163:0/3668797216 wait complete. 2024-03-11T22:57:52.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:52.630+0000 7fce5c59f700 1 Processor -- start 2024-03-11T22:57:52.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:52.630+0000 7fce5c59f700 1 -- start start 2024-03-11T22:57:52.635 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:52.630+0000 7fce5c59f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce540fcef0 0x7fce5410db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:57:52.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:52.630+0000 7fce5c59f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce540fe0e0 0x7fce5410e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:57:52.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:52.630+0000 7fce5c59f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5410e660 con 0x7fce540fe0e0 2024-03-11T22:57:52.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:52.630+0000 7fce5c59f700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5410ef00 con 0x7fce540fcef0 2024-03-11T22:57:52.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:52.630+0000 7fce5a33b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce540fcef0 0x7fce5410db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:57:52.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:52.630+0000 7fce5a33b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce540fcef0 0x7fce5410db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:59620/0 (socket says 172.21.15.163:59620) 2024-03-11T22:57:52.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:52.630+0000 7fce5a33b700 1 -- 172.21.15.163:0/3095596878 learned_addr learned my addr 172.21.15.163:0/3095596878 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-03-11T22:57:52.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:52.630+0000 7fce59b3a700 1 --2- 172.21.15.163:0/3095596878 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce540fe0e0 0x7fce5410e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:57:52.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:52.630+0000 7fce5a33b700 1 -- 172.21.15.163:0/3095596878 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce540fe0e0 msgr2=0x7fce5410e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:57:52.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:52.630+0000 7fce5a33b700 1 --2- 172.21.15.163:0/3095596878 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce540fe0e0 0x7fce5410e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:57:52.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:52.630+0000 7fce5a33b700 1 -- 172.21.15.163:0/3095596878 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce50009940 con 0x7fce540fcef0 2024-03-11T22:57:52.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:52.631+0000 7fce5a33b700 1 --2- 172.21.15.163:0/3095596878 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce540fcef0 0x7fce5410db20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fce5000b700 tx=0x7fce50004d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:57:52.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:52.631+0000 7fce4affd700 1 -- 172.21.15.163:0/3095596878 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce5000e040 con 0x7fce540fcef0 2024-03-11T22:57:52.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:52.631+0000 7fce4affd700 1 -- 172.21.15.163:0/3095596878 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce500049b0 con 0x7fce540fcef0 2024-03-11T22:57:52.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:52.631+0000 7fce4affd700 1 -- 172.21.15.163:0/3095596878 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce5001c9e0 con 0x7fce540fcef0 2024-03-11T22:57:52.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:52.631+0000 7fce5c59f700 1 -- 172.21.15.163:0/3095596878 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce5410f180 con 0x7fce540fcef0 2024-03-11T22:57:52.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:52.631+0000 7fce5c59f700 1 -- 172.21.15.163:0/3095596878 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce5410f6c0 con 0x7fce540fcef0 2024-03-11T22:57:52.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:52.632+0000 7fce5c59f700 1 -- 172.21.15.163:0/3095596878 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce5404bed0 con 0x7fce540fcef0 2024-03-11T22:57:52.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:52.632+0000 7fce4affd700 1 -- 172.21.15.163:0/3095596878 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fce50012410 con 0x7fce540fcef0 2024-03-11T22:57:52.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:52.633+0000 7fce4affd700 1 --2- 172.21.15.163:0/3095596878 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fce400692b0 0x7fce4006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:57:52.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:52.633+0000 7fce4affd700 1 -- 172.21.15.163:0/3095596878 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 3770+0+0 (secure 0 0 0) 0x7fce50088800 con 0x7fce540fcef0 2024-03-11T22:57:52.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:52.633+0000 7fce59b3a700 1 --2- 172.21.15.163:0/3095596878 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fce400692b0 0x7fce4006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:57:52.638 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:52.633+0000 7fce59b3a700 1 --2- 172.21.15.163:0/3095596878 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fce400692b0 0x7fce4006b760 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fce44004f40 tx=0x7fce4400b040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:57:52.641 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:52.637+0000 7fce4affd700 1 -- 172.21.15.163:0/3095596878 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7fce50058be0 con 0x7fce540fcef0 2024-03-11T22:57:52.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:52 smithi153 ceph-mon[28962]: osdmap e30: 4 total, 4 up, 4 in 2024-03-11T22:57:52.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:52 smithi153 ceph-mon[28962]: Detected new or changed devices on smithi163 2024-03-11T22:57:52.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:52 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:52.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:52 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:52.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:52 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:57:52.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:52 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:52.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:52 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:52.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:52 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:57:52.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:52 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:52.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:52 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:57:52.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:52 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:52.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:52 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:57:52.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:52 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:57:52.830 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:57:52.827+0000 7fce5c59f700 1 -- 172.21.15.163:0/3095596878 --> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fce54102ac0 con 0x7fce400692b0 2024-03-11T22:57:53.534 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:53 smithi163 ceph-mon[35584]: Adjusting osd_memory_target on smithi163 to 15936M 2024-03-11T22:57:53.535 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:53 smithi163 ceph-mon[35584]: pgmap v95: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-11T22:57:53.535 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:53 smithi163 ceph-mon[35584]: osdmap e31: 4 total, 4 up, 4 in 2024-03-11T22:57:53.535 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:53 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:57:53.535 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:53 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:57:53.535 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:53 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:53.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:53 smithi153 ceph-mon[28962]: Adjusting osd_memory_target on smithi163 to 15936M 2024-03-11T22:57:53.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:53 smithi153 ceph-mon[28962]: pgmap v95: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-11T22:57:53.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:53 smithi153 ceph-mon[28962]: osdmap e31: 4 total, 4 up, 4 in 2024-03-11T22:57:53.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:53 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:57:53.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:53 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:57:53.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:53 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:54.449 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:54 smithi163 ceph-mon[35584]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:57:54.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:54 smithi153 ceph-mon[28962]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:57:55.639 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:55 smithi163 ceph-mon[35584]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-03-11T22:57:55.639 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:55 smithi163 ceph-mon[35584]: from='client.? 172.21.15.163:0/1974508258' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b9caf0a9-46bf-44bc-a2d0-b5f0e1b3eadf"}]: dispatch 2024-03-11T22:57:55.639 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:55 smithi163 ceph-mon[35584]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b9caf0a9-46bf-44bc-a2d0-b5f0e1b3eadf"}]: dispatch 2024-03-11T22:57:55.640 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:55 smithi163 ceph-mon[35584]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b9caf0a9-46bf-44bc-a2d0-b5f0e1b3eadf"}]': finished 2024-03-11T22:57:55.640 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:55 smithi163 ceph-mon[35584]: osdmap e32: 5 total, 4 up, 5 in 2024-03-11T22:57:55.640 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:55 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:57:55.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:55 smithi153 ceph-mon[28962]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-03-11T22:57:55.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:55 smithi153 ceph-mon[28962]: from='client.? 172.21.15.163:0/1974508258' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b9caf0a9-46bf-44bc-a2d0-b5f0e1b3eadf"}]: dispatch 2024-03-11T22:57:55.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:55 smithi153 ceph-mon[28962]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b9caf0a9-46bf-44bc-a2d0-b5f0e1b3eadf"}]: dispatch 2024-03-11T22:57:55.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:55 smithi153 ceph-mon[28962]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b9caf0a9-46bf-44bc-a2d0-b5f0e1b3eadf"}]': finished 2024-03-11T22:57:55.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:55 smithi153 ceph-mon[28962]: osdmap e32: 5 total, 4 up, 5 in 2024-03-11T22:57:55.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:55 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:57:56.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:56 smithi163 ceph-mon[35584]: from='client.? 172.21.15.163:0/2413407866' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:57:56.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:56 smithi153 ceph-mon[28962]: from='client.? 172.21.15.163:0/2413407866' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:57:57.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:57 smithi163 ceph-mon[35584]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-03-11T22:57:57.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:57 smithi153 ceph-mon[28962]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-03-11T22:57:59.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:57:59 smithi163 ceph-mon[35584]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-11T22:57:59.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:57:59 smithi153 ceph-mon[28962]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-11T22:58:01.588 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:01 smithi163 ceph-mon[35584]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-11T22:58:01.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:01 smithi153 ceph-mon[28962]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-11T22:58:03.396 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:03 smithi163 ceph-mon[35584]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-11T22:58:03.397 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:03 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T22:58:03.397 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:03 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:58:03.397 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:03 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:58:03.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:03 smithi153 ceph-mon[28962]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-11T22:58:03.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:03 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T22:58:03.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:03 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:58:03.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:03 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:58:04.570 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:04 smithi163 ceph-mon[35584]: Deploying daemon osd.4 on smithi163 2024-03-11T22:58:04.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:04 smithi153 ceph-mon[28962]: Deploying daemon osd.4 on smithi163 2024-03-11T22:58:05.406 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:05 smithi163 ceph-mon[35584]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-11T22:58:05.681 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:05 smithi153 ceph-mon[28962]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-11T22:58:07.598 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:07 smithi163 ceph-mon[35584]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-11T22:58:07.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:07 smithi153 ceph-mon[28962]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-11T22:58:09.238 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:08 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:09.238 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:08 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:09.238 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:08 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:58:09.238 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:08 smithi163 ceph-mon[35584]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-11T22:58:09.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:08 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:09.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:08 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:09.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:08 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:58:09.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:08 smithi153 ceph-mon[28962]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-11T22:58:11.629 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:11 smithi163 ceph-mon[35584]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-11T22:58:11.695 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:11.690+0000 7fce4affd700 1 -- 172.21.15.163:0/3095596878 <== mgr.14204 v2:172.21.15.153:6800/29938766 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fce54102ac0 con 0x7fce400692b0 2024-03-11T22:58:11.696 INFO:teuthology.orchestra.run.smithi163.stdout:Created osd(s) 4 on host 'smithi163' 2024-03-11T22:58:11.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:11.693+0000 7fce5c59f700 1 -- 172.21.15.163:0/3095596878 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fce400692b0 msgr2=0x7fce4006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:58:11.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:11.693+0000 7fce5c59f700 1 --2- 172.21.15.163:0/3095596878 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fce400692b0 0x7fce4006b760 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fce44004f40 tx=0x7fce4400b040 comp rx=0 tx=0).stop 2024-03-11T22:58:11.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:11.693+0000 7fce5c59f700 1 -- 172.21.15.163:0/3095596878 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce540fcef0 msgr2=0x7fce5410db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:58:11.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:11.693+0000 7fce5c59f700 1 --2- 172.21.15.163:0/3095596878 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce540fcef0 0x7fce5410db20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fce5000b700 tx=0x7fce50004d10 comp rx=0 tx=0).stop 2024-03-11T22:58:11.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:11.693+0000 7fce5c59f700 1 -- 172.21.15.163:0/3095596878 shutdown_connections 2024-03-11T22:58:11.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:11.693+0000 7fce5c59f700 1 --2- 172.21.15.163:0/3095596878 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fce400692b0 0x7fce4006b760 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:11.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:11.693+0000 7fce5c59f700 1 --2- 172.21.15.163:0/3095596878 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce540fcef0 0x7fce5410db20 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:11.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:11.693+0000 7fce5c59f700 1 --2- 172.21.15.163:0/3095596878 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce540fe0e0 0x7fce5410e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:11.698 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:11.693+0000 7fce5c59f700 1 -- 172.21.15.163:0/3095596878 >> 172.21.15.163:0/3095596878 conn(0x7fce540f8620 msgr2=0x7fce541013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:58:11.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:11.693+0000 7fce5c59f700 1 -- 172.21.15.163:0/3095596878 shutdown_connections 2024-03-11T22:58:11.699 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:11.693+0000 7fce5c59f700 1 -- 172.21.15.163:0/3095596878 wait complete. 2024-03-11T22:58:11.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:11 smithi153 ceph-mon[28962]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-11T22:58:12.298 DEBUG:teuthology.orchestra.run.smithi163:osd.4> sudo journalctl -f -n 0 -u ceph-fa913b96-dff9-11ee-95c8-87774f69a715@osd.4.service 2024-03-11T22:58:12.300 INFO:tasks.cephadm:Deploying osd.5 on smithi163 with /dev/vg_nvme/lv_2... 2024-03-11T22:58:12.300 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-11T22:58:12.603 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi163/config 2024-03-11T22:58:12.977 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:12 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:12.978 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:12 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:12.978 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:12 smithi163 ceph-mon[35584]: from='osd.4 [v2:172.21.15.163:6808/4243752277,v1:172.21.15.163:6809/4243752277]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T22:58:12.978 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:12 smithi163 ceph-mon[35584]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T22:58:12.978 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:12 smithi163 ceph-mon[35584]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-11T22:58:12.978 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:12 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:12.978 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:12 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:13.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:12 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:13.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:12 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:13.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:12 smithi153 ceph-mon[28962]: from='osd.4 [v2:172.21.15.163:6808/4243752277,v1:172.21.15.163:6809/4243752277]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T22:58:13.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:12 smithi153 ceph-mon[28962]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T22:58:13.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:12 smithi153 ceph-mon[28962]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-11T22:58:13.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:12 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:13.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:12 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:13.682 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:13 smithi163 ceph-mon[35584]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T22:58:13.683 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:13 smithi163 ceph-mon[35584]: osdmap e33: 5 total, 4 up, 5 in 2024-03-11T22:58:13.683 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:13 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:58:13.683 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:13 smithi163 ceph-mon[35584]: from='osd.4 [v2:172.21.15.163:6808/4243752277,v1:172.21.15.163:6809/4243752277]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-03-11T22:58:13.683 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:13 smithi163 ceph-mon[35584]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-03-11T22:58:13.935 INFO:journalctl@ceph.osd.4.smithi163.stdout:Mar 11 22:58:13 smithi163 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-4[47435]: 2024-03-11T22:58:13.676+0000 7ff3fbcc7700 -1 osd.4 0 waiting for initial osdmap 2024-03-11T22:58:13.935 INFO:journalctl@ceph.osd.4.smithi163.stdout:Mar 11 22:58:13 smithi163 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-4[47435]: 2024-03-11T22:58:13.687+0000 7ff3f4abe700 -1 osd.4 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:58:14.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:13 smithi153 ceph-mon[28962]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T22:58:14.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:13 smithi153 ceph-mon[28962]: osdmap e33: 5 total, 4 up, 5 in 2024-03-11T22:58:14.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:13 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:58:14.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:13 smithi153 ceph-mon[28962]: from='osd.4 [v2:172.21.15.163:6808/4243752277,v1:172.21.15.163:6809/4243752277]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-03-11T22:58:14.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:13 smithi153 ceph-mon[28962]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-03-11T22:58:14.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:14 smithi163 ceph-mon[35584]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-03-11T22:58:14.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:14 smithi163 ceph-mon[35584]: osdmap e34: 5 total, 4 up, 5 in 2024-03-11T22:58:14.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:14 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:58:14.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:14 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:58:14.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:14 smithi163 ceph-mon[35584]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-11T22:58:15.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:14 smithi153 ceph-mon[28962]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-03-11T22:58:15.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:14 smithi153 ceph-mon[28962]: osdmap e34: 5 total, 4 up, 5 in 2024-03-11T22:58:15.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:14 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:58:15.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:14 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:58:15.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:14 smithi153 ceph-mon[28962]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-11T22:58:15.577 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-11T22:58:15.577 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-11T22:58:15.577 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10+0 records in 2024-03-11T22:58:15.577 INFO:teuthology.orchestra.run.smithi163.stderr:10+0 records out 2024-03-11T22:58:15.578 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0133585 s, 785 MB/s 2024-03-11T22:58:15.578 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping successful for: 2024-03-11T22:58:15.981 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:15 smithi163 ceph-mon[35584]: purged_snaps scrub starts 2024-03-11T22:58:15.981 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:15 smithi163 ceph-mon[35584]: purged_snaps scrub ok 2024-03-11T22:58:15.981 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:15 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:58:15.981 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:15 smithi163 ceph-mon[35584]: osd.4 [v2:172.21.15.163:6808/4243752277,v1:172.21.15.163:6809/4243752277] boot 2024-03-11T22:58:15.981 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:15 smithi163 ceph-mon[35584]: osdmap e35: 5 total, 5 up, 5 in 2024-03-11T22:58:15.982 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:15 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:58:16.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:15 smithi153 ceph-mon[28962]: purged_snaps scrub starts 2024-03-11T22:58:16.034 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:15 smithi153 ceph-mon[28962]: purged_snaps scrub ok 2024-03-11T22:58:16.034 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:15 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:58:16.034 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:15 smithi153 ceph-mon[28962]: osd.4 [v2:172.21.15.163:6808/4243752277,v1:172.21.15.163:6809/4243752277] boot 2024-03-11T22:58:16.034 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:15 smithi153 ceph-mon[28962]: osdmap e35: 5 total, 5 up, 5 in 2024-03-11T22:58:16.034 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:15 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T22:58:16.273 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi163:vg_nvme/lv_2 2024-03-11T22:58:16.904 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:16 smithi163 ceph-mon[35584]: osdmap e36: 5 total, 5 up, 5 in 2024-03-11T22:58:16.904 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:16 smithi163 ceph-mon[35584]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-11T22:58:16.980 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi163/config 2024-03-11T22:58:17.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:16 smithi153 ceph-mon[28962]: osdmap e36: 5 total, 5 up, 5 in 2024-03-11T22:58:17.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:16 smithi153 ceph-mon[28962]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-11T22:58:18.238 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:17 smithi163 ceph-mon[35584]: Detected new or changed devices on smithi163 2024-03-11T22:58:18.238 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:17 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:18.238 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:17 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:18.239 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:17 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:58:18.239 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:17 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:58:18.239 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:17 smithi163 ceph-mon[35584]: Adjusting osd_memory_target on smithi163 to 7968M 2024-03-11T22:58:18.239 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:17 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:18.239 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:17 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:58:18.239 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:17 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:58:18.239 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:17 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:18.239 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:17 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:58:18.239 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:17 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:58:18.239 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:17 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:58:18.239 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:17 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:18.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:17 smithi153 ceph-mon[28962]: Detected new or changed devices on smithi163 2024-03-11T22:58:18.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:17 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:18.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:17 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:18.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:17 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:58:18.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:17 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:58:18.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:17 smithi153 ceph-mon[28962]: Adjusting osd_memory_target on smithi163 to 7968M 2024-03-11T22:58:18.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:17 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:18.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:17 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:58:18.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:17 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:58:18.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:17 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:18.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:17 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:58:18.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:17 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:58:18.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:17 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:58:18.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:17 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:18.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:18.625+0000 7fad98b13700 1 -- 172.21.15.163:0/885624110 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fad940fbe40 msgr2=0x7fad940fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:58:18.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:18.625+0000 7fad98b13700 1 --2- 172.21.15.163:0/885624110 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fad940fbe40 0x7fad940fc250 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fad88004940 tx=0x7fad88009e30 comp rx=0 tx=0).stop 2024-03-11T22:58:18.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:18.625+0000 7fad98b13700 1 -- 172.21.15.163:0/885624110 shutdown_connections 2024-03-11T22:58:18.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:18.625+0000 7fad98b13700 1 --2- 172.21.15.163:0/885624110 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad940fcfb0 0x7fad940fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:18.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:18.625+0000 7fad98b13700 1 --2- 172.21.15.163:0/885624110 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fad940fbe40 0x7fad940fc250 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:18.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:18.625+0000 7fad98b13700 1 -- 172.21.15.163:0/885624110 >> 172.21.15.163:0/885624110 conn(0x7fad940f7520 msgr2=0x7fad940f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:58:18.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:18.625+0000 7fad98b13700 1 -- 172.21.15.163:0/885624110 shutdown_connections 2024-03-11T22:58:18.631 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:18.625+0000 7fad98b13700 1 -- 172.21.15.163:0/885624110 wait complete. 2024-03-11T22:58:18.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:18.626+0000 7fad98b13700 1 Processor -- start 2024-03-11T22:58:18.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:18.626+0000 7fad98b13700 1 -- start start 2024-03-11T22:58:18.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:18.626+0000 7fad98b13700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fad940fbe40 0x7fad9410d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:58:18.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:18.626+0000 7fad98b13700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad940fcfb0 0x7fad9410dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:58:18.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:18.626+0000 7fad98b13700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad9410e530 con 0x7fad940fcfb0 2024-03-11T22:58:18.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:18.626+0000 7fad98b13700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad9410ed70 con 0x7fad940fbe40 2024-03-11T22:58:18.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:18.626+0000 7fad9259c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fad940fbe40 0x7fad9410d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:58:18.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:18.626+0000 7fad9259c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fad940fbe40 0x7fad9410d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:47312/0 (socket says 172.21.15.163:47312) 2024-03-11T22:58:18.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:18.626+0000 7fad9259c700 1 -- 172.21.15.163:0/910933206 learned_addr learned my addr 172.21.15.163:0/910933206 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-03-11T22:58:18.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:18.626+0000 7fad9259c700 1 -- 172.21.15.163:0/910933206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad940fcfb0 msgr2=0x7fad9410dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:58:18.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:18.626+0000 7fad9259c700 1 --2- 172.21.15.163:0/910933206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad940fcfb0 0x7fad9410dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:18.632 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:18.626+0000 7fad9259c700 1 -- 172.21.15.163:0/910933206 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad88005040 con 0x7fad940fbe40 2024-03-11T22:58:18.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:18.626+0000 7fad9259c700 1 --2- 172.21.15.163:0/910933206 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fad940fbe40 0x7fad9410d960 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fad880093b0 tx=0x7fad88009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:58:18.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:18.627+0000 7fad82ffd700 1 -- 172.21.15.163:0/910933206 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad8800e040 con 0x7fad940fbe40 2024-03-11T22:58:18.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:18.627+0000 7fad98b13700 1 -- 172.21.15.163:0/910933206 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad9410eff0 con 0x7fad940fbe40 2024-03-11T22:58:18.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:18.627+0000 7fad82ffd700 1 -- 172.21.15.163:0/910933206 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fad8800bcc0 con 0x7fad940fbe40 2024-03-11T22:58:18.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:18.627+0000 7fad82ffd700 1 -- 172.21.15.163:0/910933206 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad8801c630 con 0x7fad940fbe40 2024-03-11T22:58:18.633 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:18.627+0000 7fad98b13700 1 -- 172.21.15.163:0/910933206 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad9410f530 con 0x7fad940fbe40 2024-03-11T22:58:18.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:18.628+0000 7fad98b13700 1 -- 172.21.15.163:0/910933206 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad9404bed0 con 0x7fad940fbe40 2024-03-11T22:58:18.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:18.628+0000 7fad82ffd700 1 -- 172.21.15.163:0/910933206 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fad880139f0 con 0x7fad940fbe40 2024-03-11T22:58:18.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:18.628+0000 7fad82ffd700 1 --2- 172.21.15.163:0/910933206 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fad7c0692a0 0x7fad7c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:58:18.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:18.629+0000 7fad82ffd700 1 -- 172.21.15.163:0/910933206 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4202+0+0 (secure 0 0 0) 0x7fad880880a0 con 0x7fad940fbe40 2024-03-11T22:58:18.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:18.629+0000 7fad91d9b700 1 --2- 172.21.15.163:0/910933206 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fad7c0692a0 0x7fad7c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:58:18.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:18.629+0000 7fad91d9b700 1 --2- 172.21.15.163:0/910933206 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fad7c0692a0 0x7fad7c06b750 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fad84006fd0 tx=0x7fad84006db0 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:58:18.636 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:18.633+0000 7fad82ffd700 1 -- 172.21.15.163:0/910933206 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7fad88058350 con 0x7fad940fbe40 2024-03-11T22:58:18.826 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:18.822+0000 7fad98b13700 1 -- 172.21.15.163:0/910933206 --> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fad94066660 con 0x7fad7c0692a0 2024-03-11T22:58:19.238 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:18 smithi163 ceph-mon[35584]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-11T22:58:19.238 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:18 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:58:19.238 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:18 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:58:19.238 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:18 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:58:19.238 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:18 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:58:19.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:18 smithi153 ceph-mon[28962]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-11T22:58:19.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:18 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:58:19.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:18 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:58:19.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:18 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:58:19.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:18 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:58:20.466 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:20 smithi163 ceph-mon[35584]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:58:20.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:20 smithi153 ceph-mon[28962]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:58:21.653 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:21 smithi163 ceph-mon[35584]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-11T22:58:21.654 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:21 smithi163 ceph-mon[35584]: from='client.? 172.21.15.163:0/776324679' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4e2dc8c-98c7-4be5-a0a2-917ec54e5910"}]: dispatch 2024-03-11T22:58:21.654 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:21 smithi163 ceph-mon[35584]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4e2dc8c-98c7-4be5-a0a2-917ec54e5910"}]: dispatch 2024-03-11T22:58:21.654 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:21 smithi163 ceph-mon[35584]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b4e2dc8c-98c7-4be5-a0a2-917ec54e5910"}]': finished 2024-03-11T22:58:21.654 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:21 smithi163 ceph-mon[35584]: osdmap e37: 6 total, 5 up, 6 in 2024-03-11T22:58:21.654 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:21 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:58:21.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:21 smithi153 ceph-mon[28962]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-11T22:58:21.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:21 smithi153 ceph-mon[28962]: from='client.? 172.21.15.163:0/776324679' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4e2dc8c-98c7-4be5-a0a2-917ec54e5910"}]: dispatch 2024-03-11T22:58:21.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:21 smithi153 ceph-mon[28962]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4e2dc8c-98c7-4be5-a0a2-917ec54e5910"}]: dispatch 2024-03-11T22:58:21.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:21 smithi153 ceph-mon[28962]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b4e2dc8c-98c7-4be5-a0a2-917ec54e5910"}]': finished 2024-03-11T22:58:21.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:21 smithi153 ceph-mon[28962]: osdmap e37: 6 total, 5 up, 6 in 2024-03-11T22:58:21.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:21 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:58:22.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:22 smithi163 ceph-mon[35584]: from='client.? 172.21.15.163:0/1821264573' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:58:22.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:22 smithi153 ceph-mon[28962]: from='client.? 172.21.15.163:0/1821264573' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:58:23.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:23 smithi163 ceph-mon[35584]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-11T22:58:23.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:23 smithi153 ceph-mon[28962]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-11T22:58:25.470 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:25 smithi163 ceph-mon[35584]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-11T22:58:25.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:25 smithi153 ceph-mon[28962]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-11T22:58:27.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:27 smithi163 ceph-mon[35584]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-11T22:58:27.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:27 smithi153 ceph-mon[28962]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-11T22:58:29.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:29 smithi163 ceph-mon[35584]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-11T22:58:29.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:29 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T22:58:29.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:29 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:58:29.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:29 smithi163 ceph-mon[35584]: Deploying daemon osd.5 on smithi163 2024-03-11T22:58:29.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:29 smithi153 ceph-mon[28962]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-11T22:58:29.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:29 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T22:58:29.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:29 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:58:29.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:29 smithi153 ceph-mon[28962]: Deploying daemon osd.5 on smithi163 2024-03-11T22:58:31.679 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:31 smithi163 ceph-mon[35584]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-11T22:58:31.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:31 smithi153 ceph-mon[28962]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-11T22:58:33.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:33 smithi163 ceph-mon[35584]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-11T22:58:33.489 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:33 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:58:33.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:33 smithi153 ceph-mon[28962]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-11T22:58:33.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:33 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:58:35.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:34 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:35.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:34 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:35.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:34 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:58:35.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:34 smithi153 ceph-mon[28962]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-11T22:58:35.072 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:34 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:35.073 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:34 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:35.073 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:34 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:58:35.073 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:34 smithi163 ceph-mon[35584]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-11T22:58:36.837 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:36.833+0000 7fad82ffd700 1 -- 172.21.15.163:0/910933206 <== mgr.14204 v2:172.21.15.153:6800/29938766 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fad94066660 con 0x7fad7c0692a0 2024-03-11T22:58:36.837 INFO:teuthology.orchestra.run.smithi163.stdout:Created osd(s) 5 on host 'smithi163' 2024-03-11T22:58:36.839 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:36.835+0000 7fad98b13700 1 -- 172.21.15.163:0/910933206 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fad7c0692a0 msgr2=0x7fad7c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:58:36.839 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:36.835+0000 7fad98b13700 1 --2- 172.21.15.163:0/910933206 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fad7c0692a0 0x7fad7c06b750 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fad84006fd0 tx=0x7fad84006db0 comp rx=0 tx=0).stop 2024-03-11T22:58:36.839 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:36.835+0000 7fad98b13700 1 -- 172.21.15.163:0/910933206 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fad940fbe40 msgr2=0x7fad9410d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:58:36.840 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:36.835+0000 7fad98b13700 1 --2- 172.21.15.163:0/910933206 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fad940fbe40 0x7fad9410d960 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fad880093b0 tx=0x7fad88009e30 comp rx=0 tx=0).stop 2024-03-11T22:58:36.840 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:36.835+0000 7fad98b13700 1 -- 172.21.15.163:0/910933206 shutdown_connections 2024-03-11T22:58:36.840 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:36.835+0000 7fad98b13700 1 --2- 172.21.15.163:0/910933206 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fad7c0692a0 0x7fad7c06b750 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:36.840 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:36.835+0000 7fad98b13700 1 --2- 172.21.15.163:0/910933206 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fad940fbe40 0x7fad9410d960 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:36.840 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:36.835+0000 7fad98b13700 1 --2- 172.21.15.163:0/910933206 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad940fcfb0 0x7fad9410dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:36.840 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:36.835+0000 7fad98b13700 1 -- 172.21.15.163:0/910933206 >> 172.21.15.163:0/910933206 conn(0x7fad940f7520 msgr2=0x7fad94065f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:58:36.840 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:36.835+0000 7fad98b13700 1 -- 172.21.15.163:0/910933206 shutdown_connections 2024-03-11T22:58:36.840 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:36.835+0000 7fad98b13700 1 -- 172.21.15.163:0/910933206 wait complete. 2024-03-11T22:58:37.404 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:37 smithi163 ceph-mon[35584]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-11T22:58:37.404 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:37 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:37.404 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:37 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:37.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:37 smithi153 ceph-mon[28962]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-11T22:58:37.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:37 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:37.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:37 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:37.703 DEBUG:teuthology.orchestra.run.smithi163:osd.5> sudo journalctl -f -n 0 -u ceph-fa913b96-dff9-11ee-95c8-87774f69a715@osd.5.service 2024-03-11T22:58:37.705 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-03-11T22:58:37.706 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph osd stat -f json 2024-03-11T22:58:37.966 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:58:38.238 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:38 smithi163 ceph-mon[35584]: from='osd.5 [v2:172.21.15.163:6816/2486862776,v1:172.21.15.163:6817/2486862776]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T22:58:38.238 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:38 smithi163 ceph-mon[35584]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T22:58:38.369 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:38 smithi153 ceph-mon[28962]: from='osd.5 [v2:172.21.15.163:6816/2486862776,v1:172.21.15.163:6817/2486862776]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T22:58:38.370 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:38 smithi153 ceph-mon[28962]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T22:58:39.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:39 smithi153 ceph-mon[28962]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-11T22:58:39.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:39 smithi153 ceph-mon[28962]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T22:58:39.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:39 smithi153 ceph-mon[28962]: osdmap e38: 6 total, 5 up, 6 in 2024-03-11T22:58:39.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:39 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:58:39.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:39 smithi153 ceph-mon[28962]: from='osd.5 [v2:172.21.15.163:6816/2486862776,v1:172.21.15.163:6817/2486862776]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-03-11T22:58:39.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:39 smithi153 ceph-mon[28962]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-03-11T22:58:39.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:39 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:39.282 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:39 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:39.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:39 smithi163 ceph-mon[35584]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-11T22:58:39.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:39 smithi163 ceph-mon[35584]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T22:58:39.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:39 smithi163 ceph-mon[35584]: osdmap e38: 6 total, 5 up, 6 in 2024-03-11T22:58:39.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:39 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:58:39.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:39 smithi163 ceph-mon[35584]: from='osd.5 [v2:172.21.15.163:6816/2486862776,v1:172.21.15.163:6817/2486862776]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-03-11T22:58:39.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:39 smithi163 ceph-mon[35584]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-03-11T22:58:39.427 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:39 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:39.428 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:39 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:39.428 INFO:journalctl@ceph.osd.5.smithi163.stdout:Mar 11 22:58:39 smithi163 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-5[54367]: 2024-03-11T22:58:39.148+0000 7f1b02671700 -1 osd.5 0 waiting for initial osdmap 2024-03-11T22:58:39.428 INFO:journalctl@ceph.osd.5.smithi163.stdout:Mar 11 22:58:39 smithi163 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-5[54367]: 2024-03-11T22:58:39.156+0000 7f1afc46a700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:58:39.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.503+0000 7fe0b921e700 1 -- 172.21.15.153:0/2934520763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0b40fe080 msgr2=0x7fe0b40fe4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:58:39.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.503+0000 7fe0b921e700 1 --2- 172.21.15.153:0/2934520763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0b40fe080 0x7fe0b40fe4f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe0a8004940 tx=0x7fe0a8009e30 comp rx=0 tx=0).stop 2024-03-11T22:58:39.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.503+0000 7fe0b921e700 1 -- 172.21.15.153:0/2934520763 shutdown_connections 2024-03-11T22:58:39.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.503+0000 7fe0b921e700 1 --2- 172.21.15.153:0/2934520763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0b40fe080 0x7fe0b40fe4f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:39.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.503+0000 7fe0b921e700 1 --2- 172.21.15.153:0/2934520763 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe0b40fce80 0x7fe0b40fd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:39.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.503+0000 7fe0b921e700 1 -- 172.21.15.153:0/2934520763 >> 172.21.15.153:0/2934520763 conn(0x7fe0b40f8550 msgr2=0x7fe0b40fa9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:58:39.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.503+0000 7fe0b921e700 1 -- 172.21.15.153:0/2934520763 shutdown_connections 2024-03-11T22:58:39.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.503+0000 7fe0b921e700 1 -- 172.21.15.153:0/2934520763 wait complete. 2024-03-11T22:58:39.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.504+0000 7fe0b921e700 1 Processor -- start 2024-03-11T22:58:39.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.504+0000 7fe0b921e700 1 -- start start 2024-03-11T22:58:39.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.504+0000 7fe0b921e700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0b40fce80 0x7fe0b410d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:58:39.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.504+0000 7fe0b921e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe0b40fe080 0x7fe0b410dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:58:39.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.504+0000 7fe0b921e700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b410e380 con 0x7fe0b40fce80 2024-03-11T22:58:39.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.504+0000 7fe0b921e700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b410ec20 con 0x7fe0b40fe080 2024-03-11T22:58:39.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.504+0000 7fe0b2d9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0b40fce80 0x7fe0b410d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:58:39.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.504+0000 7fe0b2d9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0b40fce80 0x7fe0b410d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:37892/0 (socket says 172.21.15.153:37892) 2024-03-11T22:58:39.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.504+0000 7fe0b2d9d700 1 -- 172.21.15.153:0/1127309927 learned_addr learned my addr 172.21.15.153:0/1127309927 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:58:39.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.504+0000 7fe0b259c700 1 --2- 172.21.15.153:0/1127309927 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe0b40fe080 0x7fe0b410dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:58:39.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.505+0000 7fe0b2d9d700 1 -- 172.21.15.153:0/1127309927 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe0b40fe080 msgr2=0x7fe0b410dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:58:39.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.505+0000 7fe0b2d9d700 1 --2- 172.21.15.153:0/1127309927 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe0b40fe080 0x7fe0b410dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:39.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.505+0000 7fe0b2d9d700 1 -- 172.21.15.153:0/1127309927 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a8005040 con 0x7fe0b40fce80 2024-03-11T22:58:39.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.505+0000 7fe0b2d9d700 1 --2- 172.21.15.153:0/1127309927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0b40fce80 0x7fe0b410d840 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe0a400c8b0 tx=0x7fe0a4004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:58:39.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.505+0000 7fe09b7fe700 1 -- 172.21.15.153:0/1127309927 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0a40106f0 con 0x7fe0b40fce80 2024-03-11T22:58:39.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.505+0000 7fe09b7fe700 1 -- 172.21.15.153:0/1127309927 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe0a4010d20 con 0x7fe0b40fce80 2024-03-11T22:58:39.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.505+0000 7fe0b921e700 1 -- 172.21.15.153:0/1127309927 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0b410ef00 con 0x7fe0b40fce80 2024-03-11T22:58:39.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.505+0000 7fe09b7fe700 1 -- 172.21.15.153:0/1127309927 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe0a4012e90 con 0x7fe0b40fce80 2024-03-11T22:58:39.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.505+0000 7fe0b921e700 1 -- 172.21.15.153:0/1127309927 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0b410f440 con 0x7fe0b40fce80 2024-03-11T22:58:39.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.506+0000 7fe0b921e700 1 -- 172.21.15.153:0/1127309927 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0b404bed0 con 0x7fe0b40fce80 2024-03-11T22:58:39.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.506+0000 7fe09b7fe700 1 -- 172.21.15.153:0/1127309927 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fe0a4010850 con 0x7fe0b40fce80 2024-03-11T22:58:39.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.507+0000 7fe09b7fe700 1 --2- 172.21.15.153:0/1127309927 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fe09c0692a0 0x7fe09c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:58:39.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.507+0000 7fe0b259c700 1 --2- 172.21.15.153:0/1127309927 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fe09c0692a0 0x7fe09c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:58:39.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.507+0000 7fe09b7fe700 1 -- 172.21.15.153:0/1127309927 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4354+0+0 (secure 0 0 0) 0x7fe0a4086ce0 con 0x7fe0b40fce80 2024-03-11T22:58:39.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.507+0000 7fe0b259c700 1 --2- 172.21.15.153:0/1127309927 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fe09c0692a0 0x7fe09c06b750 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe0a800b3b0 tx=0x7fe0a8009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:58:39.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.511+0000 7fe09b7fe700 1 -- 172.21.15.153:0/1127309927 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7fe0a4056e70 con 0x7fe0b40fce80 2024-03-11T22:58:39.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.692+0000 7fe0b921e700 1 -- 172.21.15.153:0/1127309927 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fe0b4113e60 con 0x7fe0b40fce80 2024-03-11T22:58:39.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.692+0000 7fe09b7fe700 1 -- 172.21.15.153:0/1127309927 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v39) v1 ==== 74+0+130 (secure 0 0 0) 0x7fe0a4056a00 con 0x7fe0b40fce80 2024-03-11T22:58:39.697 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:58:39.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.694+0000 7fe0b921e700 1 -- 172.21.15.153:0/1127309927 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fe09c0692a0 msgr2=0x7fe09c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:58:39.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.694+0000 7fe0b921e700 1 --2- 172.21.15.153:0/1127309927 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fe09c0692a0 0x7fe09c06b750 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe0a800b3b0 tx=0x7fe0a8009e30 comp rx=0 tx=0).stop 2024-03-11T22:58:39.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.694+0000 7fe0b921e700 1 -- 172.21.15.153:0/1127309927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0b40fce80 msgr2=0x7fe0b410d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:58:39.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.694+0000 7fe0b921e700 1 --2- 172.21.15.153:0/1127309927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0b40fce80 0x7fe0b410d840 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe0a400c8b0 tx=0x7fe0a4004470 comp rx=0 tx=0).stop 2024-03-11T22:58:39.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.694+0000 7fe0b921e700 1 -- 172.21.15.153:0/1127309927 shutdown_connections 2024-03-11T22:58:39.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.694+0000 7fe0b921e700 1 --2- 172.21.15.153:0/1127309927 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe0b40fce80 0x7fe0b410d840 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:39.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.694+0000 7fe0b921e700 1 --2- 172.21.15.153:0/1127309927 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fe09c0692a0 0x7fe09c06b750 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:39.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.694+0000 7fe0b921e700 1 --2- 172.21.15.153:0/1127309927 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe0b40fe080 0x7fe0b410dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:39.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.694+0000 7fe0b921e700 1 -- 172.21.15.153:0/1127309927 >> 172.21.15.153:0/1127309927 conn(0x7fe0b40f8550 msgr2=0x7fe0b4101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:58:39.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.694+0000 7fe0b921e700 1 -- 172.21.15.153:0/1127309927 shutdown_connections 2024-03-11T22:58:39.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:39.694+0000 7fe0b921e700 1 -- 172.21.15.153:0/1127309927 wait complete. 2024-03-11T22:58:40.381 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:40 smithi153 ceph-mon[28962]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-03-11T22:58:40.381 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:40 smithi153 ceph-mon[28962]: osdmap e39: 6 total, 5 up, 6 in 2024-03-11T22:58:40.381 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:40 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:58:40.381 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:40 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:58:40.381 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:40 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/1127309927' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T22:58:40.405 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":39,"num_osds":6,"num_up_osds":5,"osd_up_since":1710197894,"num_in_osds":6,"osd_in_since":1710197901,"num_remapped_pgs":0} 2024-03-11T22:58:40.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:40 smithi163 ceph-mon[35584]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-03-11T22:58:40.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:40 smithi163 ceph-mon[35584]: osdmap e39: 6 total, 5 up, 6 in 2024-03-11T22:58:40.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:40 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:58:40.489 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:40 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:58:40.489 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:40 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/1127309927' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T22:58:41.406 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph osd stat -f json 2024-03-11T22:58:41.433 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:41 smithi153 ceph-mon[28962]: purged_snaps scrub starts 2024-03-11T22:58:41.433 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:41 smithi153 ceph-mon[28962]: purged_snaps scrub ok 2024-03-11T22:58:41.433 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:41 smithi153 ceph-mon[28962]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-11T22:58:41.433 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:41 smithi153 ceph-mon[28962]: osd.5 [v2:172.21.15.163:6816/2486862776,v1:172.21.15.163:6817/2486862776] boot 2024-03-11T22:58:41.433 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:41 smithi153 ceph-mon[28962]: osdmap e40: 6 total, 6 up, 6 in 2024-03-11T22:58:41.433 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:41 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:58:41.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:41 smithi163 ceph-mon[35584]: purged_snaps scrub starts 2024-03-11T22:58:41.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:41 smithi163 ceph-mon[35584]: purged_snaps scrub ok 2024-03-11T22:58:41.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:41 smithi163 ceph-mon[35584]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-11T22:58:41.489 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:41 smithi163 ceph-mon[35584]: osd.5 [v2:172.21.15.163:6816/2486862776,v1:172.21.15.163:6817/2486862776] boot 2024-03-11T22:58:41.489 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:41 smithi163 ceph-mon[35584]: osdmap e40: 6 total, 6 up, 6 in 2024-03-11T22:58:41.489 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T22:58:41.669 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:58:42.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:42 smithi153 ceph-mon[28962]: osdmap e41: 6 total, 6 up, 6 in 2024-03-11T22:58:42.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:42 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:42.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:42 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:42.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:42 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:58:42.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:42 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:58:42.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:42 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:58:42.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:42 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:42.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:42 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:58:42.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:42 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:58:42.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:42 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:42.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:42 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:58:42.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:42 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:58:42.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:42 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:58:42.532 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:42 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:42.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:42 smithi163 ceph-mon[35584]: osdmap e41: 6 total, 6 up, 6 in 2024-03-11T22:58:42.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:42 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:42.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:42 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:42.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:42 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:58:42.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:42 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:58:42.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:42 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:58:42.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:42 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:42.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:42 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:58:42.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:42 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:58:42.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:42 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:42.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:42 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:58:42.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:42 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:58:42.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:42 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:58:42.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:42 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:58:43.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.400+0000 7f4bb2df9700 1 -- 172.21.15.153:0/3349779450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bac0fd470 msgr2=0x7f4bac0fd880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:58:43.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.400+0000 7f4bb2df9700 1 --2- 172.21.15.153:0/3349779450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bac0fd470 0x7f4bac0fd880 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4ba0004940 tx=0x7f4ba0009e30 comp rx=0 tx=0).stop 2024-03-11T22:58:43.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.400+0000 7f4bb2df9700 1 -- 172.21.15.153:0/3349779450 shutdown_connections 2024-03-11T22:58:43.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.400+0000 7f4bb2df9700 1 --2- 172.21.15.153:0/3349779450 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4bac0fc180 0x7f4bac0fc5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:43.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.400+0000 7f4bb2df9700 1 --2- 172.21.15.153:0/3349779450 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bac0fd470 0x7f4bac0fd880 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:43.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.400+0000 7f4bb2df9700 1 -- 172.21.15.153:0/3349779450 >> 172.21.15.153:0/3349779450 conn(0x7f4bac0f6ca0 msgr2=0x7f4bac0f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:58:43.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.401+0000 7f4bb2df9700 1 -- 172.21.15.153:0/3349779450 shutdown_connections 2024-03-11T22:58:43.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.401+0000 7f4bb2df9700 1 -- 172.21.15.153:0/3349779450 wait complete. 2024-03-11T22:58:43.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.401+0000 7f4bb2df9700 1 Processor -- start 2024-03-11T22:58:43.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.401+0000 7f4bb2df9700 1 -- start start 2024-03-11T22:58:43.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.402+0000 7f4bb2df9700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4bac0fc180 0x7f4bac10c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:58:43.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.402+0000 7f4bb2df9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bac0fd470 0x7f4bac10ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:58:43.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.402+0000 7f4bb2df9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bac10d490 con 0x7f4bac0fd470 2024-03-11T22:58:43.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.402+0000 7f4bb2df9700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bac10dd30 con 0x7f4bac0fc180 2024-03-11T22:58:43.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.402+0000 7f4babfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bac0fd470 0x7f4bac10ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:58:43.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.402+0000 7f4babfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bac0fd470 0x7f4bac10ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47012/0 (socket says 172.21.15.153:47012) 2024-03-11T22:58:43.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.402+0000 7f4babfff700 1 -- 172.21.15.153:0/618244777 learned_addr learned my addr 172.21.15.153:0/618244777 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:58:43.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.402+0000 7f4bb0b95700 1 --2- 172.21.15.153:0/618244777 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4bac0fc180 0x7f4bac10c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:58:43.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.402+0000 7f4babfff700 1 -- 172.21.15.153:0/618244777 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4bac0fc180 msgr2=0x7f4bac10c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:58:43.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.402+0000 7f4babfff700 1 --2- 172.21.15.153:0/618244777 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4bac0fc180 0x7f4bac10c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:43.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.402+0000 7f4babfff700 1 -- 172.21.15.153:0/618244777 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba0005040 con 0x7f4bac0fd470 2024-03-11T22:58:43.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.402+0000 7f4babfff700 1 --2- 172.21.15.153:0/618244777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bac0fd470 0x7f4bac10ce90 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4b9800c830 tx=0x7f4b98004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:58:43.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.402+0000 7f4ba97fa700 1 -- 172.21.15.153:0/618244777 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b980106f0 con 0x7f4bac0fd470 2024-03-11T22:58:43.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.402+0000 7f4ba97fa700 1 -- 172.21.15.153:0/618244777 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4b98010d20 con 0x7f4bac0fd470 2024-03-11T22:58:43.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.402+0000 7f4bb2df9700 1 -- 172.21.15.153:0/618244777 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bac10e010 con 0x7f4bac0fd470 2024-03-11T22:58:43.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.402+0000 7f4ba97fa700 1 -- 172.21.15.153:0/618244777 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b980113a0 con 0x7f4bac0fd470 2024-03-11T22:58:43.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.402+0000 7f4bb2df9700 1 -- 172.21.15.153:0/618244777 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bac10e550 con 0x7f4bac0fd470 2024-03-11T22:58:43.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.403+0000 7f4bb2df9700 1 -- 172.21.15.153:0/618244777 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bac04bed0 con 0x7f4bac0fd470 2024-03-11T22:58:43.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.404+0000 7f4ba97fa700 1 -- 172.21.15.153:0/618244777 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f4b98012400 con 0x7f4bac0fd470 2024-03-11T22:58:43.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.404+0000 7f4ba97fa700 1 --2- 172.21.15.153:0/618244777 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f4b9c0692a0 0x7f4b9c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:58:43.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.404+0000 7f4ba97fa700 1 -- 172.21.15.153:0/618244777 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4b98086df0 con 0x7f4bac0fd470 2024-03-11T22:58:43.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.404+0000 7f4bb0b95700 1 --2- 172.21.15.153:0/618244777 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f4b9c0692a0 0x7f4b9c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:58:43.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.405+0000 7f4bb0b95700 1 --2- 172.21.15.153:0/618244777 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f4b9c0692a0 0x7f4b9c06b750 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4ba000b3b0 tx=0x7f4ba0009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:58:43.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.409+0000 7f4ba97fa700 1 -- 172.21.15.153:0/618244777 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f4b98056ef0 con 0x7f4bac0fd470 2024-03-11T22:58:43.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:43 smithi153 ceph-mon[28962]: Detected new or changed devices on smithi163 2024-03-11T22:58:43.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:43 smithi153 ceph-mon[28962]: Adjusting osd_memory_target on smithi163 to 5312M 2024-03-11T22:58:43.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:43 smithi153 ceph-mon[28962]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:58:43.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.595+0000 7f4bb2df9700 1 -- 172.21.15.153:0/618244777 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f4bac104960 con 0x7f4bac0fd470 2024-03-11T22:58:43.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.596+0000 7f4ba97fa700 1 -- 172.21.15.153:0/618244777 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v41) v1 ==== 74+0+130 (secure 0 0 0) 0x7f4b98056a80 con 0x7f4bac0fd470 2024-03-11T22:58:43.600 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:58:43.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.598+0000 7f4bb2df9700 1 -- 172.21.15.153:0/618244777 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f4b9c0692a0 msgr2=0x7f4b9c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:58:43.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.598+0000 7f4bb2df9700 1 --2- 172.21.15.153:0/618244777 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f4b9c0692a0 0x7f4b9c06b750 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4ba000b3b0 tx=0x7f4ba0009e30 comp rx=0 tx=0).stop 2024-03-11T22:58:43.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.598+0000 7f4bb2df9700 1 -- 172.21.15.153:0/618244777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bac0fd470 msgr2=0x7f4bac10ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:58:43.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.598+0000 7f4bb2df9700 1 --2- 172.21.15.153:0/618244777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bac0fd470 0x7f4bac10ce90 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4b9800c830 tx=0x7f4b98004470 comp rx=0 tx=0).stop 2024-03-11T22:58:43.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.598+0000 7f4bb2df9700 1 -- 172.21.15.153:0/618244777 shutdown_connections 2024-03-11T22:58:43.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.598+0000 7f4bb2df9700 1 --2- 172.21.15.153:0/618244777 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f4b9c0692a0 0x7f4b9c06b750 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:43.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.598+0000 7f4bb2df9700 1 --2- 172.21.15.153:0/618244777 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4bac0fc180 0x7f4bac10c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:43.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.598+0000 7f4bb2df9700 1 --2- 172.21.15.153:0/618244777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4bac0fd470 0x7f4bac10ce90 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:43.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.598+0000 7f4bb2df9700 1 -- 172.21.15.153:0/618244777 >> 172.21.15.153:0/618244777 conn(0x7f4bac0f6ca0 msgr2=0x7f4bac1003c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:58:43.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.598+0000 7f4bb2df9700 1 -- 172.21.15.153:0/618244777 shutdown_connections 2024-03-11T22:58:43.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:43.598+0000 7f4bb2df9700 1 -- 172.21.15.153:0/618244777 wait complete. 2024-03-11T22:58:43.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:43 smithi163 ceph-mon[35584]: Detected new or changed devices on smithi163 2024-03-11T22:58:43.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:43 smithi163 ceph-mon[35584]: Adjusting osd_memory_target on smithi163 to 5312M 2024-03-11T22:58:43.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:43 smithi163 ceph-mon[35584]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:58:44.203 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":41,"num_osds":6,"num_up_osds":6,"osd_up_since":1710197920,"num_in_osds":6,"osd_in_since":1710197901,"num_remapped_pgs":0} 2024-03-11T22:58:44.204 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T22:58:44.508 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:58:44.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:44 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/618244777' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T22:58:44.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:44 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/618244777' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T22:58:45.594 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:45 smithi153 ceph-mon[28962]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:58:45.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:45 smithi163 ceph-mon[35584]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:58:46.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.064+0000 7f64d8cf1700 1 -- 172.21.15.153:0/1364965413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64d4102f40 msgr2=0x7f64d4103310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:58:46.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.064+0000 7f64d8cf1700 1 --2- 172.21.15.153:0/1364965413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64d4102f40 0x7f64d4103310 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f64cc009ce0 tx=0x7f64cc004d10 comp rx=0 tx=0).stop 2024-03-11T22:58:46.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.064+0000 7f64d8cf1700 1 -- 172.21.15.153:0/1364965413 shutdown_connections 2024-03-11T22:58:46.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.064+0000 7f64d8cf1700 1 --2- 172.21.15.153:0/1364965413 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64d41038e0 0x7f64d40fcf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:46.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.064+0000 7f64d8cf1700 1 --2- 172.21.15.153:0/1364965413 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64d4102f40 0x7f64d4103310 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:46.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.064+0000 7f64d8cf1700 1 -- 172.21.15.153:0/1364965413 >> 172.21.15.153:0/1364965413 conn(0x7f64d40f8af0 msgr2=0x7f64d40faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:58:46.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.064+0000 7f64d8cf1700 1 -- 172.21.15.153:0/1364965413 shutdown_connections 2024-03-11T22:58:46.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.064+0000 7f64d8cf1700 1 -- 172.21.15.153:0/1364965413 wait complete. 2024-03-11T22:58:46.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.065+0000 7f64d8cf1700 1 Processor -- start 2024-03-11T22:58:46.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.065+0000 7f64d8cf1700 1 -- start start 2024-03-11T22:58:46.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.065+0000 7f64d8cf1700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64d4102f40 0x7f64d4113bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:58:46.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.065+0000 7f64d8cf1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64d41038e0 0x7f64d4114110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:58:46.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.065+0000 7f64d8cf1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d4114760 con 0x7f64d41038e0 2024-03-11T22:58:46.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.065+0000 7f64d8cf1700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d410da90 con 0x7f64d4102f40 2024-03-11T22:58:46.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.065+0000 7f64d1d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64d41038e0 0x7f64d4114110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:58:46.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.065+0000 7f64d1d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64d41038e0 0x7f64d4114110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47038/0 (socket says 172.21.15.153:47038) 2024-03-11T22:58:46.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.065+0000 7f64d1d9b700 1 -- 172.21.15.153:0/731995071 learned_addr learned my addr 172.21.15.153:0/731995071 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:58:46.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.065+0000 7f64d259c700 1 --2- 172.21.15.153:0/731995071 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64d4102f40 0x7f64d4113bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:58:46.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.065+0000 7f64d1d9b700 1 -- 172.21.15.153:0/731995071 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64d4102f40 msgr2=0x7f64d4113bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:58:46.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.065+0000 7f64d1d9b700 1 --2- 172.21.15.153:0/731995071 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64d4102f40 0x7f64d4113bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:46.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.065+0000 7f64d1d9b700 1 -- 172.21.15.153:0/731995071 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64cc009940 con 0x7f64d41038e0 2024-03-11T22:58:46.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.066+0000 7f64d1d9b700 1 --2- 172.21.15.153:0/731995071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64d41038e0 0x7f64d4114110 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f64c800c890 tx=0x7f64c80044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:58:46.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.066+0000 7f64c2ffd700 1 -- 172.21.15.153:0/731995071 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64c800a700 con 0x7f64d41038e0 2024-03-11T22:58:46.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.066+0000 7f64c2ffd700 1 -- 172.21.15.153:0/731995071 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f64c800ad30 con 0x7f64d41038e0 2024-03-11T22:58:46.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.066+0000 7f64d8cf1700 1 -- 172.21.15.153:0/731995071 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64d410dd70 con 0x7f64d41038e0 2024-03-11T22:58:46.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.066+0000 7f64c2ffd700 1 -- 172.21.15.153:0/731995071 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f64c80113a0 con 0x7f64d41038e0 2024-03-11T22:58:46.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.066+0000 7f64d8cf1700 1 -- 172.21.15.153:0/731995071 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64d410e2b0 con 0x7f64d41038e0 2024-03-11T22:58:46.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.067+0000 7f64d8cf1700 1 -- 172.21.15.153:0/731995071 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64d4100250 con 0x7f64d41038e0 2024-03-11T22:58:46.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.067+0000 7f64c2ffd700 1 -- 172.21.15.153:0/731995071 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f64c8012400 con 0x7f64d41038e0 2024-03-11T22:58:46.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.068+0000 7f64c2ffd700 1 --2- 172.21.15.153:0/731995071 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f64bc0692a0 0x7f64bc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:58:46.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.068+0000 7f64d259c700 1 --2- 172.21.15.153:0/731995071 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f64bc0692a0 0x7f64bc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:58:46.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.068+0000 7f64c2ffd700 1 -- 172.21.15.153:0/731995071 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f64c8086f00 con 0x7f64d41038e0 2024-03-11T22:58:46.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.068+0000 7f64d259c700 1 --2- 172.21.15.153:0/731995071 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f64bc0692a0 0x7f64bc06b750 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f64cc015040 tx=0x7f64cc004d10 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:58:46.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.072+0000 7f64c2ffd700 1 -- 172.21.15.153:0/731995071 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f64c8057000 con 0x7f64d41038e0 2024-03-11T22:58:46.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.258+0000 7f64d8cf1700 1 -- 172.21.15.153:0/731995071 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64d410a040 con 0x7f64d41038e0 2024-03-11T22:58:46.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.259+0000 7f64c2ffd700 1 -- 172.21.15.153:0/731995071 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11145 (secure 0 0 0) 0x7f64c8056b90 con 0x7f64d41038e0 2024-03-11T22:58:46.263 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:58:46.264 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":41,"fsid":"fa913b96-dff9-11ee-95c8-87774f69a715","created":"2024-03-11T22:53:14.269171+0000","modified":"2024-03-11T22:58:41.287239+0000","last_up_change":"2024-03-11T22:58:40.147211+0000","last_in_change":"2024-03-11T22:58:21.268596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T22:57:26.368808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"542731eb-3fb3-450b-93d6-c6a8a9df1659","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1406194697},{"type":"v1","addr":"172.21.15.153:6803","nonce":1406194697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1406194697},{"type":"v1","addr":"172.21.15.153:6805","nonce":1406194697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1406194697},{"type":"v1","addr":"172.21.15.153:6809","nonce":1406194697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1406194697},{"type":"v1","addr":"172.21.15.153:6807","nonce":1406194697}]},"public_addr":"172.21.15.153:6803/1406194697","cluster_addr":"172.21.15.153:6805/1406194697","heartbeat_back_addr":"172.21.15.153:6809/1406194697","heartbeat_front_addr":"172.21.15.153:6807/1406194697","state":["exists","up"]},{"osd":1,"uuid":"7be06d22-d83a-4b39-ace8-3d800bbd1aee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2718652372},{"type":"v1","addr":"172.21.15.153:6811","nonce":2718652372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2718652372},{"type":"v1","addr":"172.21.15.153:6813","nonce":2718652372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2718652372},{"type":"v1","addr":"172.21.15.153:6817","nonce":2718652372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2718652372},{"type":"v1","addr":"172.21.15.153:6815","nonce":2718652372}]},"public_addr":"172.21.15.153:6811/2718652372","cluster_addr":"172.21.15.153:6813/2718652372","heartbeat_back_addr":"172.21.15.153:6817/2718652372","heartbeat_front_addr":"172.21.15.153:6815/2718652372","state":["exists","up"]},{"osd":2,"uuid":"b4863162-3a3d-45ae-9bc0-c8b069ff063a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":4016153774},{"type":"v1","addr":"172.21.15.153:6819","nonce":4016153774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":4016153774},{"type":"v1","addr":"172.21.15.153:6821","nonce":4016153774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4016153774},{"type":"v1","addr":"172.21.15.153:6825","nonce":4016153774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":4016153774},{"type":"v1","addr":"172.21.15.153:6823","nonce":4016153774}]},"public_addr":"172.21.15.153:6819/4016153774","cluster_addr":"172.21.15.153:6821/4016153774","heartbeat_back_addr":"172.21.15.153:6825/4016153774","heartbeat_front_addr":"172.21.15.153:6823/4016153774","state":["exists","up"]},{"osd":3,"uuid":"92a62b43-e245-4d71-b4ca-89a7d518af50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":3768791983},{"type":"v1","addr":"172.21.15.163:6801","nonce":3768791983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":3768791983},{"type":"v1","addr":"172.21.15.163:6803","nonce":3768791983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":3768791983},{"type":"v1","addr":"172.21.15.163:6807","nonce":3768791983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":3768791983},{"type":"v1","addr":"172.21.15.163:6805","nonce":3768791983}]},"public_addr":"172.21.15.163:6801/3768791983","cluster_addr":"172.21.15.163:6803/3768791983","heartbeat_back_addr":"172.21.15.163:6807/3768791983","heartbeat_front_addr":"172.21.15.163:6805/3768791983","state":["exists","up"]},{"osd":4,"uuid":"b9caf0a9-46bf-44bc-a2d0-b5f0e1b3eadf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":4243752277},{"type":"v1","addr":"172.21.15.163:6809","nonce":4243752277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":4243752277},{"type":"v1","addr":"172.21.15.163:6811","nonce":4243752277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":4243752277},{"type":"v1","addr":"172.21.15.163:6815","nonce":4243752277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":4243752277},{"type":"v1","addr":"172.21.15.163:6813","nonce":4243752277}]},"public_addr":"172.21.15.163:6809/4243752277","cluster_addr":"172.21.15.163:6811/4243752277","heartbeat_back_addr":"172.21.15.163:6815/4243752277","heartbeat_front_addr":"172.21.15.163:6813/4243752277","state":["exists","up"]},{"osd":5,"uuid":"b4e2dc8c-98c7-4be5-a0a2-917ec54e5910","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2486862776},{"type":"v1","addr":"172.21.15.163:6817","nonce":2486862776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2486862776},{"type":"v1","addr":"172.21.15.163:6819","nonce":2486862776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2486862776},{"type":"v1","addr":"172.21.15.163:6823","nonce":2486862776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2486862776},{"type":"v1","addr":"172.21.15.163:6821","nonce":2486862776}]},"public_addr":"172.21.15.163:6817/2486862776","cluster_addr":"172.21.15.163:6819/2486862776","heartbeat_back_addr":"172.21.15.163:6823/2486862776","heartbeat_front_addr":"172.21.15.163:6821/2486862776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:56:27.613880+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:56:55.525968+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:57:22.785587+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:57:47.270057+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:58:12.791831+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:58:38.648113+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/1864158862":"2024-03-12T22:55:18.043211+0000","172.21.15.153:0/41949006":"2024-03-12T22:55:18.043211+0000","172.21.15.153:6801/4212724556":"2024-03-12T22:55:18.043211+0000","172.21.15.153:0/369153497":"2024-03-12T22:55:18.043211+0000","172.21.15.153:6800/4212724556":"2024-03-12T22:55:18.043211+0000","172.21.15.153:6801/1518383615":"2024-03-12T22:54:17.065945+0000","172.21.15.153:6800/1518383615":"2024-03-12T22:54:17.065945+0000","172.21.15.153:0/1974568910":"2024-03-12T22:54:17.065945+0000","172.21.15.153:0/2653190270":"2024-03-12T22:54:17.065945+0000","172.21.15.153:6800/2578439373":"2024-03-12T22:53:46.835889+0000","172.21.15.153:6801/2578439373":"2024-03-12T22:53:46.835889+0000","172.21.15.153:0/3587313628":"2024-03-12T22:53:46.835889+0000","172.21.15.153:0/3414319920":"2024-03-12T22:54:17.065945+0000","172.21.15.153:0/3612064784":"2024-03-12T22:53:46.835889+0000","172.21.15.153:0/3032487131":"2024-03-12T22:53:46.835889+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T22:58:46.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.261+0000 7f64d8cf1700 1 -- 172.21.15.153:0/731995071 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f64bc0692a0 msgr2=0x7f64bc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:58:46.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.261+0000 7f64d8cf1700 1 --2- 172.21.15.153:0/731995071 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f64bc0692a0 0x7f64bc06b750 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f64cc015040 tx=0x7f64cc004d10 comp rx=0 tx=0).stop 2024-03-11T22:58:46.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.261+0000 7f64d8cf1700 1 -- 172.21.15.153:0/731995071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64d41038e0 msgr2=0x7f64d4114110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:58:46.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.261+0000 7f64d8cf1700 1 --2- 172.21.15.153:0/731995071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64d41038e0 0x7f64d4114110 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f64c800c890 tx=0x7f64c80044c0 comp rx=0 tx=0).stop 2024-03-11T22:58:46.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.261+0000 7f64d8cf1700 1 -- 172.21.15.153:0/731995071 shutdown_connections 2024-03-11T22:58:46.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.261+0000 7f64d8cf1700 1 --2- 172.21.15.153:0/731995071 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f64bc0692a0 0x7f64bc06b750 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:46.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.261+0000 7f64d8cf1700 1 --2- 172.21.15.153:0/731995071 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64d4102f40 0x7f64d4113bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:46.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.261+0000 7f64d8cf1700 1 --2- 172.21.15.153:0/731995071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64d41038e0 0x7f64d4114110 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:46.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.261+0000 7f64d8cf1700 1 -- 172.21.15.153:0/731995071 >> 172.21.15.153:0/731995071 conn(0x7f64d40f8af0 msgr2=0x7f64d40fa500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:58:46.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.261+0000 7f64d8cf1700 1 -- 172.21.15.153:0/731995071 shutdown_connections 2024-03-11T22:58:46.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:46.261+0000 7f64d8cf1700 1 -- 172.21.15.153:0/731995071 wait complete. 2024-03-11T22:58:46.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:46 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/731995071' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:58:46.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:46 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/731995071' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:58:46.842 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T22:57:26.368808+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-03-11T22:58:46.842 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-11T22:58:47.100 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:58:47.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:47 smithi153 ceph-mon[28962]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:58:47.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:47 smithi163 ceph-mon[35584]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:58:48.462 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:48 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:58:48.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:48 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:58:48.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.650+0000 7f29b4fe5700 1 -- 172.21.15.153:0/1416842546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29b00fdfe0 msgr2=0x7f29b00fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:58:48.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.650+0000 7f29b4fe5700 1 --2- 172.21.15.153:0/1416842546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29b00fdfe0 0x7f29b00fe3b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f29a0004970 tx=0x7f29a0009e30 comp rx=0 tx=0).stop 2024-03-11T22:58:48.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.650+0000 7f29b4fe5700 1 -- 172.21.15.153:0/1416842546 shutdown_connections 2024-03-11T22:58:48.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.650+0000 7f29b4fe5700 1 --2- 172.21.15.153:0/1416842546 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29b00fe8f0 0x7f29b0107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:48.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.650+0000 7f29b4fe5700 1 --2- 172.21.15.153:0/1416842546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29b00fdfe0 0x7f29b00fe3b0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:48.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.650+0000 7f29b4fe5700 1 -- 172.21.15.153:0/1416842546 >> 172.21.15.153:0/1416842546 conn(0x7f29b0071d90 msgr2=0x7f29b0072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:58:48.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.650+0000 7f29b4fe5700 1 -- 172.21.15.153:0/1416842546 shutdown_connections 2024-03-11T22:58:48.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.650+0000 7f29b4fe5700 1 -- 172.21.15.153:0/1416842546 wait complete. 2024-03-11T22:58:48.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.650+0000 7f29b4fe5700 1 Processor -- start 2024-03-11T22:58:48.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.650+0000 7f29b4fe5700 1 -- start start 2024-03-11T22:58:48.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.651+0000 7f29b4fe5700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29b00fdfe0 0x7f29b00fbea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:58:48.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.651+0000 7f29b4fe5700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29b00fe8f0 0x7f29b00fc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:58:48.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.651+0000 7f29b4fe5700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b0111940 con 0x7f29b00fdfe0 2024-03-11T22:58:48.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.651+0000 7f29b4fe5700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b0111ab0 con 0x7f29b00fe8f0 2024-03-11T22:58:48.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.651+0000 7f29ae59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29b00fdfe0 0x7f29b00fbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:58:48.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.651+0000 7f29ae59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29b00fdfe0 0x7f29b00fbea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47056/0 (socket says 172.21.15.153:47056) 2024-03-11T22:58:48.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.651+0000 7f29ae59c700 1 -- 172.21.15.153:0/291802994 learned_addr learned my addr 172.21.15.153:0/291802994 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:58:48.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.651+0000 7f29a7fff700 1 --2- 172.21.15.153:0/291802994 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29b00fe8f0 0x7f29b00fc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:58:48.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.651+0000 7f29ae59c700 1 -- 172.21.15.153:0/291802994 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29b00fe8f0 msgr2=0x7f29b00fc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:58:48.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.651+0000 7f29ae59c700 1 --2- 172.21.15.153:0/291802994 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29b00fe8f0 0x7f29b00fc3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:48.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.651+0000 7f29ae59c700 1 -- 172.21.15.153:0/291802994 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a0005040 con 0x7f29b00fdfe0 2024-03-11T22:58:48.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.651+0000 7f29ae59c700 1 --2- 172.21.15.153:0/291802994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29b00fdfe0 0x7f29b00fbea0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f29a0000c00 tx=0x7f29a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:58:48.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.651+0000 7f29a6ffd700 1 -- 172.21.15.153:0/291802994 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29a000e040 con 0x7f29b00fdfe0 2024-03-11T22:58:48.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.651+0000 7f29b4fe5700 1 -- 172.21.15.153:0/291802994 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29b00fca10 con 0x7f29b00fdfe0 2024-03-11T22:58:48.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.651+0000 7f29a6ffd700 1 -- 172.21.15.153:0/291802994 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f29a0009af0 con 0x7f29b00fdfe0 2024-03-11T22:58:48.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.651+0000 7f29a6ffd700 1 -- 172.21.15.153:0/291802994 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29a001c3f0 con 0x7f29b00fdfe0 2024-03-11T22:58:48.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.651+0000 7f29b4fe5700 1 -- 172.21.15.153:0/291802994 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29b011e640 con 0x7f29b00fdfe0 2024-03-11T22:58:48.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.652+0000 7f29b4fe5700 1 -- 172.21.15.153:0/291802994 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29b01046c0 con 0x7f29b00fdfe0 2024-03-11T22:58:48.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.653+0000 7f29a6ffd700 1 -- 172.21.15.153:0/291802994 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f29a000bc10 con 0x7f29b00fdfe0 2024-03-11T22:58:48.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.653+0000 7f29a6ffd700 1 --2- 172.21.15.153:0/291802994 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f299c0692a0 0x7f299c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:58:48.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.653+0000 7f29a6ffd700 1 -- 172.21.15.153:0/291802994 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f29a00881f0 con 0x7f29b00fdfe0 2024-03-11T22:58:48.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.653+0000 7f29a7fff700 1 --2- 172.21.15.153:0/291802994 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f299c0692a0 0x7f299c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:58:48.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.654+0000 7f29a7fff700 1 --2- 172.21.15.153:0/291802994 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f299c0692a0 0x7f299c06b750 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f29b00fd8a0 tx=0x7f299800b040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:58:48.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.658+0000 7f29a6ffd700 1 -- 172.21.15.153:0/291802994 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f29a00547e0 con 0x7f29b00fdfe0 2024-03-11T22:58:48.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.844+0000 7f29b4fe5700 1 -- 172.21.15.153:0/291802994 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f29b00fd5e0 con 0x7f29b00fdfe0 2024-03-11T22:58:48.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.844+0000 7f29a6ffd700 1 -- 172.21.15.153:0/291802994 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7f29a0057e00 con 0x7f29b00fdfe0 2024-03-11T22:58:48.848 INFO:teuthology.orchestra.run.smithi153.stdout:pg_num: 1 2024-03-11T22:58:48.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.846+0000 7f29b4fe5700 1 -- 172.21.15.153:0/291802994 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f299c0692a0 msgr2=0x7f299c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:58:48.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.846+0000 7f29b4fe5700 1 --2- 172.21.15.153:0/291802994 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f299c0692a0 0x7f299c06b750 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f29b00fd8a0 tx=0x7f299800b040 comp rx=0 tx=0).stop 2024-03-11T22:58:48.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.846+0000 7f29b4fe5700 1 -- 172.21.15.153:0/291802994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29b00fdfe0 msgr2=0x7f29b00fbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:58:48.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.846+0000 7f29b4fe5700 1 --2- 172.21.15.153:0/291802994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29b00fdfe0 0x7f29b00fbea0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f29a0000c00 tx=0x7f29a0009e30 comp rx=0 tx=0).stop 2024-03-11T22:58:48.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.846+0000 7f29b4fe5700 1 -- 172.21.15.153:0/291802994 shutdown_connections 2024-03-11T22:58:48.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.846+0000 7f29b4fe5700 1 --2- 172.21.15.153:0/291802994 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f29b00fdfe0 0x7f29b00fbea0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:48.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.846+0000 7f29b4fe5700 1 --2- 172.21.15.153:0/291802994 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f299c0692a0 0x7f299c06b750 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:48.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.846+0000 7f29b4fe5700 1 --2- 172.21.15.153:0/291802994 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f29b00fe8f0 0x7f29b00fc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:48.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.846+0000 7f29b4fe5700 1 -- 172.21.15.153:0/291802994 >> 172.21.15.153:0/291802994 conn(0x7f29b0071d90 msgr2=0x7f29b0101840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:58:48.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.846+0000 7f29b4fe5700 1 -- 172.21.15.153:0/291802994 shutdown_connections 2024-03-11T22:58:48.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:48.846+0000 7f29b4fe5700 1 -- 172.21.15.153:0/291802994 wait complete. 2024-03-11T22:58:49.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:49 smithi153 ceph-mon[28962]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:58:49.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:49 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/291802994' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T22:58:49.581 INFO:tasks.cephadm:Setting up client nodes... 2024-03-11T22:58:49.581 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T22:58:49.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:49 smithi163 ceph-mon[35584]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:58:49.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:49 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/291802994' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T22:58:49.837 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:58:51.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.393+0000 7fee8bc3c700 1 -- 172.21.15.153:0/3473854265 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee840fcfb0 msgr2=0x7fee840fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:58:51.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.393+0000 7fee8bc3c700 1 --2- 172.21.15.153:0/3473854265 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee840fcfb0 0x7fee840fd420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fee74004940 tx=0x7fee74009e30 comp rx=0 tx=0).stop 2024-03-11T22:58:51.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.393+0000 7fee8bc3c700 1 -- 172.21.15.153:0/3473854265 shutdown_connections 2024-03-11T22:58:51.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.393+0000 7fee8bc3c700 1 --2- 172.21.15.153:0/3473854265 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee840fcfb0 0x7fee840fd420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:51.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.393+0000 7fee8bc3c700 1 --2- 172.21.15.153:0/3473854265 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fee840fbe40 0x7fee840fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:51.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.393+0000 7fee8bc3c700 1 -- 172.21.15.153:0/3473854265 >> 172.21.15.153:0/3473854265 conn(0x7fee840f74e0 msgr2=0x7fee840f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:58:51.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.393+0000 7fee8bc3c700 1 -- 172.21.15.153:0/3473854265 shutdown_connections 2024-03-11T22:58:51.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.393+0000 7fee8bc3c700 1 -- 172.21.15.153:0/3473854265 wait complete. 2024-03-11T22:58:51.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.394+0000 7fee8bc3c700 1 Processor -- start 2024-03-11T22:58:51.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.394+0000 7fee8bc3c700 1 -- start start 2024-03-11T22:58:51.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.394+0000 7fee8bc3c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee840fbe40 0x7fee8410d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:58:51.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.394+0000 7fee8bc3c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fee840fcfb0 0x7fee8410de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:58:51.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.394+0000 7fee8bc3c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8410e4f0 con 0x7fee840fbe40 2024-03-11T22:58:51.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.394+0000 7fee8bc3c700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8410ed30 con 0x7fee840fcfb0 2024-03-11T22:58:51.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.394+0000 7fee899d8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee840fbe40 0x7fee8410d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:58:51.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.394+0000 7fee899d8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee840fbe40 0x7fee8410d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47072/0 (socket says 172.21.15.153:47072) 2024-03-11T22:58:51.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.394+0000 7fee899d8700 1 -- 172.21.15.153:0/3977691182 learned_addr learned my addr 172.21.15.153:0/3977691182 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:58:51.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.394+0000 7fee891d7700 1 --2- 172.21.15.153:0/3977691182 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fee840fcfb0 0x7fee8410de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:58:51.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.394+0000 7fee899d8700 1 -- 172.21.15.153:0/3977691182 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fee840fcfb0 msgr2=0x7fee8410de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:58:51.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.394+0000 7fee899d8700 1 --2- 172.21.15.153:0/3977691182 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fee840fcfb0 0x7fee8410de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:51.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.394+0000 7fee899d8700 1 -- 172.21.15.153:0/3977691182 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee74005040 con 0x7fee840fbe40 2024-03-11T22:58:51.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.394+0000 7fee899d8700 1 --2- 172.21.15.153:0/3977691182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee840fbe40 0x7fee8410d920 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fee8000b790 tx=0x7fee80009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:58:51.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.395+0000 7fee7a7fc700 1 -- 172.21.15.153:0/3977691182 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee8000f6f0 con 0x7fee840fbe40 2024-03-11T22:58:51.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.395+0000 7fee7a7fc700 1 -- 172.21.15.153:0/3977691182 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fee8000fd20 con 0x7fee840fbe40 2024-03-11T22:58:51.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.395+0000 7fee8bc3c700 1 -- 172.21.15.153:0/3977691182 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee8410f010 con 0x7fee840fbe40 2024-03-11T22:58:51.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.395+0000 7fee7a7fc700 1 -- 172.21.15.153:0/3977691182 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fee800103a0 con 0x7fee840fbe40 2024-03-11T22:58:51.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.395+0000 7fee8bc3c700 1 -- 172.21.15.153:0/3977691182 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee8410f550 con 0x7fee840fbe40 2024-03-11T22:58:51.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.396+0000 7fee8bc3c700 1 -- 172.21.15.153:0/3977691182 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee8404bed0 con 0x7fee840fbe40 2024-03-11T22:58:51.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.396+0000 7fee7a7fc700 1 -- 172.21.15.153:0/3977691182 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fee80011400 con 0x7fee840fbe40 2024-03-11T22:58:51.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.396+0000 7fee7a7fc700 1 --2- 172.21.15.153:0/3977691182 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fee70069380 0x7fee7006b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:58:51.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.397+0000 7fee7a7fc700 1 -- 172.21.15.153:0/3977691182 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fee80086c10 con 0x7fee840fbe40 2024-03-11T22:58:51.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.397+0000 7fee891d7700 1 --2- 172.21.15.153:0/3977691182 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fee70069380 0x7fee7006b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:58:51.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.397+0000 7fee891d7700 1 --2- 172.21.15.153:0/3977691182 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fee70069380 0x7fee7006b830 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fee7400b7a0 tx=0x7fee74009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:58:51.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.401+0000 7fee7a7fc700 1 -- 172.21.15.153:0/3977691182 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7fee80056cf0 con 0x7fee840fbe40 2024-03-11T22:58:51.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:51 smithi153 ceph-mon[28962]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:58:51.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.660+0000 7fee8bc3c700 1 -- 172.21.15.153:0/3977691182 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fee8410f1a0 con 0x7fee840fbe40 2024-03-11T22:58:51.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.664+0000 7fee7a7fc700 1 -- 172.21.15.153:0/3977691182 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7fee8410f1a0 con 0x7fee840fbe40 2024-03-11T22:58:51.669 INFO:teuthology.orchestra.run.smithi153.stdout:[client.0] 2024-03-11T22:58:51.669 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQCrjO9lXSluJxAAYPjDIeMfcpdWBBCnmjsnBw== 2024-03-11T22:58:51.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.667+0000 7fee8bc3c700 1 -- 172.21.15.153:0/3977691182 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fee70069380 msgr2=0x7fee7006b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:58:51.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.667+0000 7fee8bc3c700 1 --2- 172.21.15.153:0/3977691182 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fee70069380 0x7fee7006b830 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fee7400b7a0 tx=0x7fee74009e30 comp rx=0 tx=0).stop 2024-03-11T22:58:51.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.667+0000 7fee8bc3c700 1 -- 172.21.15.153:0/3977691182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee840fbe40 msgr2=0x7fee8410d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:58:51.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.667+0000 7fee8bc3c700 1 --2- 172.21.15.153:0/3977691182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee840fbe40 0x7fee8410d920 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fee8000b790 tx=0x7fee80009250 comp rx=0 tx=0).stop 2024-03-11T22:58:51.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.667+0000 7fee8bc3c700 1 -- 172.21.15.153:0/3977691182 shutdown_connections 2024-03-11T22:58:51.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.667+0000 7fee8bc3c700 1 --2- 172.21.15.153:0/3977691182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fee840fbe40 0x7fee8410d920 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:51.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.667+0000 7fee8bc3c700 1 --2- 172.21.15.153:0/3977691182 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fee70069380 0x7fee7006b830 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:51.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.667+0000 7fee8bc3c700 1 --2- 172.21.15.153:0/3977691182 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fee840fcfb0 0x7fee8410de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:51.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.667+0000 7fee8bc3c700 1 -- 172.21.15.153:0/3977691182 >> 172.21.15.153:0/3977691182 conn(0x7fee840f74e0 msgr2=0x7fee840656c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:58:51.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.667+0000 7fee8bc3c700 1 -- 172.21.15.153:0/3977691182 shutdown_connections 2024-03-11T22:58:51.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:51.667+0000 7fee8bc3c700 1 -- 172.21.15.153:0/3977691182 wait complete. 2024-03-11T22:58:51.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:51 smithi163 ceph-mon[35584]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:58:52.330 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-11T22:58:52.331 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-11T22:58:52.331 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-11T22:58:52.373 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T22:58:52.639 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi163/config 2024-03-11T22:58:52.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:52 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/3977691182' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:58:52.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:52 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/3977691182' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:58:52.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:52 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/3977691182' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:58:52.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:52 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/3977691182' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:58:53.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:53 smithi163 ceph-mon[35584]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:58:53.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:53 smithi153 ceph-mon[28962]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:58:54.265 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.259+0000 7f10d1ad2700 1 -- 172.21.15.163:0/2990795170 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10cc0fcef0 msgr2=0x7f10cc0fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:58:54.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.259+0000 7f10d1ad2700 1 --2- 172.21.15.163:0/2990795170 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10cc0fcef0 0x7f10cc0fd320 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f10bc009d10 tx=0x7f10bc004d10 comp rx=0 tx=0).stop 2024-03-11T22:58:54.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.259+0000 7f10d1ad2700 1 -- 172.21.15.163:0/2990795170 shutdown_connections 2024-03-11T22:58:54.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.259+0000 7f10d1ad2700 1 --2- 172.21.15.163:0/2990795170 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10cc0fe0e0 0x7f10cc0fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:54.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.259+0000 7f10d1ad2700 1 --2- 172.21.15.163:0/2990795170 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10cc0fcef0 0x7f10cc0fd320 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:54.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.259+0000 7f10d1ad2700 1 -- 172.21.15.163:0/2990795170 >> 172.21.15.163:0/2990795170 conn(0x7f10cc0f8620 msgr2=0x7f10cc0faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:58:54.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.259+0000 7f10d1ad2700 1 -- 172.21.15.163:0/2990795170 shutdown_connections 2024-03-11T22:58:54.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.259+0000 7f10d1ad2700 1 -- 172.21.15.163:0/2990795170 wait complete. 2024-03-11T22:58:54.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.260+0000 7f10d1ad2700 1 Processor -- start 2024-03-11T22:58:54.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.260+0000 7f10d1ad2700 1 -- start start 2024-03-11T22:58:54.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.260+0000 7f10d1ad2700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10cc0fcef0 0x7f10cc10dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:58:54.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.260+0000 7f10d1ad2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10cc0fe0e0 0x7f10cc10e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:58:54.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.260+0000 7f10d1ad2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10cc10e620 con 0x7f10cc0fe0e0 2024-03-11T22:58:54.266 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.260+0000 7f10d1ad2700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10cc10eec0 con 0x7f10cc0fcef0 2024-03-11T22:58:54.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.260+0000 7f10cb7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10cc0fcef0 0x7f10cc10dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:58:54.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.260+0000 7f10cb7fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10cc0fcef0 0x7f10cc10dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:35176/0 (socket says 172.21.15.163:35176) 2024-03-11T22:58:54.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.260+0000 7f10cb7fe700 1 -- 172.21.15.163:0/848364375 learned_addr learned my addr 172.21.15.163:0/848364375 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-03-11T22:58:54.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.260+0000 7f10caffd700 1 --2- 172.21.15.163:0/848364375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10cc0fe0e0 0x7f10cc10e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:58:54.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.260+0000 7f10cb7fe700 1 -- 172.21.15.163:0/848364375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10cc0fe0e0 msgr2=0x7f10cc10e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:58:54.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.260+0000 7f10cb7fe700 1 --2- 172.21.15.163:0/848364375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10cc0fe0e0 0x7f10cc10e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:54.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.260+0000 7f10cb7fe700 1 -- 172.21.15.163:0/848364375 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10bc009940 con 0x7f10cc0fcef0 2024-03-11T22:58:54.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.261+0000 7f10cb7fe700 1 --2- 172.21.15.163:0/848364375 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10cc0fcef0 0x7f10cc10dae0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f10bc00b700 tx=0x7f10bc004d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:58:54.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.261+0000 7f10b3fff700 1 -- 172.21.15.163:0/848364375 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10bc00e040 con 0x7f10cc0fcef0 2024-03-11T22:58:54.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.261+0000 7f10d1ad2700 1 -- 172.21.15.163:0/848364375 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10cc10f140 con 0x7f10cc0fcef0 2024-03-11T22:58:54.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.261+0000 7f10b3fff700 1 -- 172.21.15.163:0/848364375 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f10bc0128e0 con 0x7f10cc0fcef0 2024-03-11T22:58:54.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.261+0000 7f10b3fff700 1 -- 172.21.15.163:0/848364375 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f10bc01c960 con 0x7f10cc0fcef0 2024-03-11T22:58:54.267 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.261+0000 7f10d1ad2700 1 -- 172.21.15.163:0/848364375 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10cc10f680 con 0x7f10cc0fcef0 2024-03-11T22:58:54.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.262+0000 7f10d1ad2700 1 -- 172.21.15.163:0/848364375 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10cc04bed0 con 0x7f10cc0fcef0 2024-03-11T22:58:54.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.262+0000 7f10b3fff700 1 -- 172.21.15.163:0/848364375 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f10bc01cbd0 con 0x7f10cc0fcef0 2024-03-11T22:58:54.268 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.262+0000 7f10b3fff700 1 --2- 172.21.15.163:0/848364375 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f10b40692a0 0x7f10b406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:58:54.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.263+0000 7f10b3fff700 1 -- 172.21.15.163:0/848364375 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f10bc088b10 con 0x7f10cc0fcef0 2024-03-11T22:58:54.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.263+0000 7f10caffd700 1 --2- 172.21.15.163:0/848364375 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f10b40692a0 0x7f10b406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:58:54.269 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.263+0000 7f10caffd700 1 --2- 172.21.15.163:0/848364375 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f10b40692a0 0x7f10b406b750 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f10c0007fd0 tx=0x7f10c0009040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:58:54.271 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.267+0000 7f10b3fff700 1 -- 172.21.15.163:0/848364375 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f10bc058b90 con 0x7f10cc0fcef0 2024-03-11T22:58:54.530 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.526+0000 7f10d1ad2700 1 -- 172.21.15.163:0/848364375 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f10cc10fbe0 con 0x7f10cc0fcef0 2024-03-11T22:58:54.535 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.531+0000 7f10b3fff700 1 -- 172.21.15.163:0/848364375 <== mon.1 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7f10bc004e10 con 0x7f10cc0fcef0 2024-03-11T22:58:54.536 INFO:teuthology.orchestra.run.smithi163.stdout:[client.1] 2024-03-11T22:58:54.536 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQCujO9ly1eDHxAAlr6zU6r81YxiXY+LcwqWYw== 2024-03-11T22:58:54.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.533+0000 7f10d1ad2700 1 -- 172.21.15.163:0/848364375 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f10b40692a0 msgr2=0x7f10b406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:58:54.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.533+0000 7f10d1ad2700 1 --2- 172.21.15.163:0/848364375 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f10b40692a0 0x7f10b406b750 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f10c0007fd0 tx=0x7f10c0009040 comp rx=0 tx=0).stop 2024-03-11T22:58:54.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.533+0000 7f10d1ad2700 1 -- 172.21.15.163:0/848364375 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10cc0fcef0 msgr2=0x7f10cc10dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:58:54.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.533+0000 7f10d1ad2700 1 --2- 172.21.15.163:0/848364375 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10cc0fcef0 0x7f10cc10dae0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f10bc00b700 tx=0x7f10bc004d10 comp rx=0 tx=0).stop 2024-03-11T22:58:54.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.533+0000 7f10d1ad2700 1 -- 172.21.15.163:0/848364375 shutdown_connections 2024-03-11T22:58:54.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.533+0000 7f10d1ad2700 1 --2- 172.21.15.163:0/848364375 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f10b40692a0 0x7f10b406b750 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:54.538 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.533+0000 7f10d1ad2700 1 --2- 172.21.15.163:0/848364375 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10cc0fcef0 0x7f10cc10dae0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:54.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.533+0000 7f10d1ad2700 1 --2- 172.21.15.163:0/848364375 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10cc0fe0e0 0x7f10cc10e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:54.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.533+0000 7f10d1ad2700 1 -- 172.21.15.163:0/848364375 >> 172.21.15.163:0/848364375 conn(0x7f10cc0f8620 msgr2=0x7f10cc1013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:58:54.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.533+0000 7f10d1ad2700 1 -- 172.21.15.163:0/848364375 shutdown_connections 2024-03-11T22:58:54.539 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-11T22:58:54.533+0000 7f10d1ad2700 1 -- 172.21.15.163:0/848364375 wait complete. 2024-03-11T22:58:55.144 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-11T22:58:55.144 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-11T22:58:55.144 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-11T22:58:55.186 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-11T22:58:55.187 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-11T22:58:55.187 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph mgr dump --format=json 2024-03-11T22:58:55.457 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:58:55.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:55 smithi163 ceph-mon[35584]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:58:55.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:55 smithi163 ceph-mon[35584]: from='client.? 172.21.15.163:0/848364375' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:58:55.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:55 smithi163 ceph-mon[35584]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:58:55.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:55 smithi163 ceph-mon[35584]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:58:55.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:55 smithi153 ceph-mon[28962]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:58:55.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:55 smithi153 ceph-mon[28962]: from='client.? 172.21.15.163:0/848364375' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:58:55.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:55 smithi153 ceph-mon[28962]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:58:55.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:55 smithi153 ceph-mon[28962]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:58:57.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.047+0000 7fc440a38700 1 -- 172.21.15.153:0/1458965537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc43c102fe0 msgr2=0x7fc43c1033b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:58:57.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.047+0000 7fc440a38700 1 --2- 172.21.15.153:0/1458965537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc43c102fe0 0x7fc43c1033b0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fc430004940 tx=0x7fc430009e30 comp rx=0 tx=0).stop 2024-03-11T22:58:57.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.047+0000 7fc440a38700 1 -- 172.21.15.153:0/1458965537 shutdown_connections 2024-03-11T22:58:57.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.047+0000 7fc440a38700 1 --2- 172.21.15.153:0/1458965537 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc43c0fb560 0x7fc43c0fb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:57.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.047+0000 7fc440a38700 1 --2- 172.21.15.153:0/1458965537 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc43c102fe0 0x7fc43c1033b0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:57.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.047+0000 7fc440a38700 1 -- 172.21.15.153:0/1458965537 >> 172.21.15.153:0/1458965537 conn(0x7fc43c071500 msgr2=0x7fc43c071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:58:57.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.047+0000 7fc440a38700 1 -- 172.21.15.153:0/1458965537 shutdown_connections 2024-03-11T22:58:57.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.047+0000 7fc440a38700 1 -- 172.21.15.153:0/1458965537 wait complete. 2024-03-11T22:58:57.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.047+0000 7fc440a38700 1 Processor -- start 2024-03-11T22:58:57.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.047+0000 7fc440a38700 1 -- start start 2024-03-11T22:58:57.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.047+0000 7fc440a38700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc43c0fb560 0x7fc43c113a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:58:57.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.047+0000 7fc440a38700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc43c102fe0 0x7fc43c113fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:58:57.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.047+0000 7fc440a38700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c114610 con 0x7fc43c0fb560 2024-03-11T22:58:57.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.047+0000 7fc440a38700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c10d940 con 0x7fc43c102fe0 2024-03-11T22:58:57.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.047+0000 7fc43a59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc43c0fb560 0x7fc43c113a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:58:57.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.047+0000 7fc43a59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc43c0fb560 0x7fc43c113a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55608/0 (socket says 172.21.15.153:55608) 2024-03-11T22:58:57.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.048+0000 7fc43a59c700 1 -- 172.21.15.153:0/263502129 learned_addr learned my addr 172.21.15.153:0/263502129 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:58:57.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.048+0000 7fc439d9b700 1 --2- 172.21.15.153:0/263502129 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc43c102fe0 0x7fc43c113fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:58:57.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.048+0000 7fc43a59c700 1 -- 172.21.15.153:0/263502129 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc43c102fe0 msgr2=0x7fc43c113fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:58:57.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.048+0000 7fc43a59c700 1 --2- 172.21.15.153:0/263502129 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc43c102fe0 0x7fc43c113fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:57.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.048+0000 7fc43a59c700 1 -- 172.21.15.153:0/263502129 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc430005040 con 0x7fc43c0fb560 2024-03-11T22:58:57.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.048+0000 7fc439d9b700 1 --2- 172.21.15.153:0/263502129 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc43c102fe0 0x7fc43c113fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T22:58:57.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.048+0000 7fc43a59c700 1 --2- 172.21.15.153:0/263502129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc43c0fb560 0x7fc43c113a80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc4300090d0 tx=0x7fc430009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:58:57.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.048+0000 7fc42affd700 1 -- 172.21.15.153:0/263502129 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc43000e040 con 0x7fc43c0fb560 2024-03-11T22:58:57.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.048+0000 7fc440a38700 1 -- 172.21.15.153:0/263502129 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc43c10dbc0 con 0x7fc43c0fb560 2024-03-11T22:58:57.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.048+0000 7fc42affd700 1 -- 172.21.15.153:0/263502129 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc43000bbf0 con 0x7fc43c0fb560 2024-03-11T22:58:57.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.048+0000 7fc42affd700 1 -- 172.21.15.153:0/263502129 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc43001c630 con 0x7fc43c0fb560 2024-03-11T22:58:57.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.048+0000 7fc440a38700 1 -- 172.21.15.153:0/263502129 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc43c10e100 con 0x7fc43c0fb560 2024-03-11T22:58:57.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.049+0000 7fc440a38700 1 -- 172.21.15.153:0/263502129 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc43c100290 con 0x7fc43c0fb560 2024-03-11T22:58:57.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.050+0000 7fc42affd700 1 -- 172.21.15.153:0/263502129 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fc43000bd60 con 0x7fc43c0fb560 2024-03-11T22:58:57.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.050+0000 7fc42affd700 1 --2- 172.21.15.153:0/263502129 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fc4240692a0 0x7fc42406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:58:57.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.050+0000 7fc42affd700 1 -- 172.21.15.153:0/263502129 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc4300881b0 con 0x7fc43c0fb560 2024-03-11T22:58:57.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.050+0000 7fc439d9b700 1 --2- 172.21.15.153:0/263502129 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fc4240692a0 0x7fc42406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:58:57.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.050+0000 7fc439d9b700 1 --2- 172.21.15.153:0/263502129 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fc4240692a0 0x7fc42406b750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc42c009fd0 tx=0x7fc42c009e50 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:58:57.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.054+0000 7fc42affd700 1 -- 172.21.15.153:0/263502129 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7fc430058360 con 0x7fc43c0fb560 2024-03-11T22:58:57.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.291+0000 7fc440a38700 1 -- 172.21.15.153:0/263502129 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fc43c04bed0 con 0x7fc43c0fb560 2024-03-11T22:58:57.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.294+0000 7fc42affd700 1 -- 172.21.15.153:0/263502129 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+167174 (secure 0 0 0) 0x7fc430013e60 con 0x7fc43c0fb560 2024-03-11T22:58:57.300 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:58:57.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.296+0000 7fc440a38700 1 -- 172.21.15.153:0/263502129 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fc4240692a0 msgr2=0x7fc42406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:58:57.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.296+0000 7fc440a38700 1 --2- 172.21.15.153:0/263502129 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fc4240692a0 0x7fc42406b750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc42c009fd0 tx=0x7fc42c009e50 comp rx=0 tx=0).stop 2024-03-11T22:58:57.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.296+0000 7fc440a38700 1 -- 172.21.15.153:0/263502129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc43c0fb560 msgr2=0x7fc43c113a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:58:57.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.296+0000 7fc440a38700 1 --2- 172.21.15.153:0/263502129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc43c0fb560 0x7fc43c113a80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc4300090d0 tx=0x7fc430009e30 comp rx=0 tx=0).stop 2024-03-11T22:58:57.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.296+0000 7fc440a38700 1 -- 172.21.15.153:0/263502129 shutdown_connections 2024-03-11T22:58:57.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.296+0000 7fc440a38700 1 --2- 172.21.15.153:0/263502129 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc43c0fb560 0x7fc43c113a80 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:57.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.296+0000 7fc440a38700 1 --2- 172.21.15.153:0/263502129 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fc4240692a0 0x7fc42406b750 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:57.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.296+0000 7fc440a38700 1 --2- 172.21.15.153:0/263502129 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc43c102fe0 0x7fc43c113fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:57.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.296+0000 7fc440a38700 1 -- 172.21.15.153:0/263502129 >> 172.21.15.153:0/263502129 conn(0x7fc43c071500 msgr2=0x7fc43c0fa530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:58:57.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.296+0000 7fc440a38700 1 -- 172.21.15.153:0/263502129 shutdown_connections 2024-03-11T22:58:57.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:57.296+0000 7fc440a38700 1 -- 172.21.15.153:0/263502129 wait complete. 2024-03-11T22:58:57.593 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:57 smithi153 ceph-mon[28962]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:58:57.593 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:57 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/263502129' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T22:58:57.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:57 smithi163 ceph-mon[35584]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:58:57.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:57 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/263502129' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T22:58:57.918 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":19,"active_gid":14204,"active_name":"smithi153.nyocep","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":29938766},{"type":"v1","addr":"172.21.15.153:6801","nonce":29938766}]},"active_addr":"172.21.15.153:6801/29938766","active_change":"2024-03-11T22:55:18.043325+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14224,"name":"smithi163.ivtqrh","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.153:8443/","prometheus":"http://172.21.15.153:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":1184875666}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":3754297840}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":1337516325}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":1661665811}]}]} 2024-03-11T22:58:57.922 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-11T22:58:57.922 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-11T22:58:57.923 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T22:58:58.181 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:58:59.610 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:58:59 smithi153 ceph-mon[28962]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:58:59.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:58:59 smithi163 ceph-mon[35584]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:58:59.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:59.830+0000 7f100eb5b700 1 -- 172.21.15.153:0/364968644 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10080702d0 msgr2=0x7f10080706a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:58:59.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:59.830+0000 7f100eb5b700 1 --2- 172.21.15.153:0/364968644 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10080702d0 0x7f10080706a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1000009d10 tx=0x7f1000004d10 comp rx=0 tx=0).stop 2024-03-11T22:58:59.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:59.830+0000 7f100eb5b700 1 -- 172.21.15.153:0/364968644 shutdown_connections 2024-03-11T22:58:59.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:59.830+0000 7f100eb5b700 1 --2- 172.21.15.153:0/364968644 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1008070be0 0x7f10081092c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:59.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:59.830+0000 7f100eb5b700 1 --2- 172.21.15.153:0/364968644 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10080702d0 0x7f10080706a0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:59.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:59.830+0000 7f100eb5b700 1 -- 172.21.15.153:0/364968644 >> 172.21.15.153:0/364968644 conn(0x7f10080f8ae0 msgr2=0x7f10080faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:58:59.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:59.830+0000 7f100eb5b700 1 -- 172.21.15.153:0/364968644 shutdown_connections 2024-03-11T22:58:59.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:59.830+0000 7f100eb5b700 1 -- 172.21.15.153:0/364968644 wait complete. 2024-03-11T22:58:59.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:59.831+0000 7f100eb5b700 1 Processor -- start 2024-03-11T22:58:59.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:59.831+0000 7f100eb5b700 1 -- start start 2024-03-11T22:58:59.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:59.831+0000 7f100eb5b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10080702d0 0x7f1008115d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:58:59.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:59.831+0000 7f100eb5b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1008070be0 0x7f10081162c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:58:59.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:59.831+0000 7f100eb5b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1008116910 con 0x7f1008070be0 2024-03-11T22:58:59.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:59.831+0000 7f100eb5b700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100810fc20 con 0x7f10080702d0 2024-03-11T22:58:59.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:59.831+0000 7f1007fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1008070be0 0x7f10081162c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:58:59.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:59.831+0000 7f1007fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1008070be0 0x7f10081162c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55626/0 (socket says 172.21.15.153:55626) 2024-03-11T22:58:59.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:59.831+0000 7f1007fff700 1 -- 172.21.15.153:0/2806916709 learned_addr learned my addr 172.21.15.153:0/2806916709 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:58:59.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:59.832+0000 7f100c8f7700 1 --2- 172.21.15.153:0/2806916709 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10080702d0 0x7f1008115d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:58:59.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:59.832+0000 7f1007fff700 1 -- 172.21.15.153:0/2806916709 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10080702d0 msgr2=0x7f1008115d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:58:59.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:59.832+0000 7f1007fff700 1 --2- 172.21.15.153:0/2806916709 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10080702d0 0x7f1008115d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:58:59.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:59.832+0000 7f1007fff700 1 -- 172.21.15.153:0/2806916709 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1000009940 con 0x7f1008070be0 2024-03-11T22:58:59.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:59.832+0000 7f100c8f7700 1 --2- 172.21.15.153:0/2806916709 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10080702d0 0x7f1008115d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T22:58:59.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:59.832+0000 7f1007fff700 1 --2- 172.21.15.153:0/2806916709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1008070be0 0x7f10081162c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc002a80 tx=0x7f0ffc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:58:59.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:59.832+0000 7f10057fa700 1 -- 172.21.15.153:0/2806916709 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ffc007cc0 con 0x7f1008070be0 2024-03-11T22:58:59.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:59.832+0000 7f10057fa700 1 -- 172.21.15.153:0/2806916709 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0ffc00f040 con 0x7f1008070be0 2024-03-11T22:58:59.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:59.832+0000 7f100eb5b700 1 -- 172.21.15.153:0/2806916709 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f100810ff00 con 0x7f1008070be0 2024-03-11T22:58:59.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:59.832+0000 7f10057fa700 1 -- 172.21.15.153:0/2806916709 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ffc0096f0 con 0x7f1008070be0 2024-03-11T22:58:59.838 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:59.832+0000 7f100eb5b700 1 -- 172.21.15.153:0/2806916709 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1008110440 con 0x7f1008070be0 2024-03-11T22:58:59.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:59.833+0000 7f100eb5b700 1 -- 172.21.15.153:0/2806916709 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1008106980 con 0x7f1008070be0 2024-03-11T22:58:59.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:59.833+0000 7f10057fa700 1 -- 172.21.15.153:0/2806916709 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f0ffc00d690 con 0x7f1008070be0 2024-03-11T22:58:59.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:59.834+0000 7f10057fa700 1 --2- 172.21.15.153:0/2806916709 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f0ff00692a0 0x7f0ff006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:58:59.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:59.834+0000 7f10057fa700 1 -- 172.21.15.153:0/2806916709 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0ffc086620 con 0x7f1008070be0 2024-03-11T22:58:59.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:59.834+0000 7f100c8f7700 1 --2- 172.21.15.153:0/2806916709 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f0ff00692a0 0x7f0ff006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:58:59.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:59.834+0000 7f100c8f7700 1 --2- 172.21.15.153:0/2806916709 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f0ff00692a0 0x7f0ff006b750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1000003400 tx=0x7f1000004d10 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:58:59.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:58:59.838+0000 7f10057fa700 1 -- 172.21.15.153:0/2806916709 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f0ffc04cd20 con 0x7f1008070be0 2024-03-11T22:59:00.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:00.020+0000 7f100eb5b700 1 -- 172.21.15.153:0/2806916709 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10080fc6e0 con 0x7f1008070be0 2024-03-11T22:59:00.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:00.021+0000 7f10057fa700 1 -- 172.21.15.153:0/2806916709 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11145 (secure 0 0 0) 0x7f0ffc0566a0 con 0x7f1008070be0 2024-03-11T22:59:00.025 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:59:00.025 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":41,"fsid":"fa913b96-dff9-11ee-95c8-87774f69a715","created":"2024-03-11T22:53:14.269171+0000","modified":"2024-03-11T22:58:41.287239+0000","last_up_change":"2024-03-11T22:58:40.147211+0000","last_in_change":"2024-03-11T22:58:21.268596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T22:57:26.368808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"542731eb-3fb3-450b-93d6-c6a8a9df1659","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1406194697},{"type":"v1","addr":"172.21.15.153:6803","nonce":1406194697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1406194697},{"type":"v1","addr":"172.21.15.153:6805","nonce":1406194697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1406194697},{"type":"v1","addr":"172.21.15.153:6809","nonce":1406194697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1406194697},{"type":"v1","addr":"172.21.15.153:6807","nonce":1406194697}]},"public_addr":"172.21.15.153:6803/1406194697","cluster_addr":"172.21.15.153:6805/1406194697","heartbeat_back_addr":"172.21.15.153:6809/1406194697","heartbeat_front_addr":"172.21.15.153:6807/1406194697","state":["exists","up"]},{"osd":1,"uuid":"7be06d22-d83a-4b39-ace8-3d800bbd1aee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2718652372},{"type":"v1","addr":"172.21.15.153:6811","nonce":2718652372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2718652372},{"type":"v1","addr":"172.21.15.153:6813","nonce":2718652372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2718652372},{"type":"v1","addr":"172.21.15.153:6817","nonce":2718652372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2718652372},{"type":"v1","addr":"172.21.15.153:6815","nonce":2718652372}]},"public_addr":"172.21.15.153:6811/2718652372","cluster_addr":"172.21.15.153:6813/2718652372","heartbeat_back_addr":"172.21.15.153:6817/2718652372","heartbeat_front_addr":"172.21.15.153:6815/2718652372","state":["exists","up"]},{"osd":2,"uuid":"b4863162-3a3d-45ae-9bc0-c8b069ff063a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":4016153774},{"type":"v1","addr":"172.21.15.153:6819","nonce":4016153774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":4016153774},{"type":"v1","addr":"172.21.15.153:6821","nonce":4016153774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4016153774},{"type":"v1","addr":"172.21.15.153:6825","nonce":4016153774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":4016153774},{"type":"v1","addr":"172.21.15.153:6823","nonce":4016153774}]},"public_addr":"172.21.15.153:6819/4016153774","cluster_addr":"172.21.15.153:6821/4016153774","heartbeat_back_addr":"172.21.15.153:6825/4016153774","heartbeat_front_addr":"172.21.15.153:6823/4016153774","state":["exists","up"]},{"osd":3,"uuid":"92a62b43-e245-4d71-b4ca-89a7d518af50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":3768791983},{"type":"v1","addr":"172.21.15.163:6801","nonce":3768791983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":3768791983},{"type":"v1","addr":"172.21.15.163:6803","nonce":3768791983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":3768791983},{"type":"v1","addr":"172.21.15.163:6807","nonce":3768791983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":3768791983},{"type":"v1","addr":"172.21.15.163:6805","nonce":3768791983}]},"public_addr":"172.21.15.163:6801/3768791983","cluster_addr":"172.21.15.163:6803/3768791983","heartbeat_back_addr":"172.21.15.163:6807/3768791983","heartbeat_front_addr":"172.21.15.163:6805/3768791983","state":["exists","up"]},{"osd":4,"uuid":"b9caf0a9-46bf-44bc-a2d0-b5f0e1b3eadf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":4243752277},{"type":"v1","addr":"172.21.15.163:6809","nonce":4243752277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":4243752277},{"type":"v1","addr":"172.21.15.163:6811","nonce":4243752277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":4243752277},{"type":"v1","addr":"172.21.15.163:6815","nonce":4243752277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":4243752277},{"type":"v1","addr":"172.21.15.163:6813","nonce":4243752277}]},"public_addr":"172.21.15.163:6809/4243752277","cluster_addr":"172.21.15.163:6811/4243752277","heartbeat_back_addr":"172.21.15.163:6815/4243752277","heartbeat_front_addr":"172.21.15.163:6813/4243752277","state":["exists","up"]},{"osd":5,"uuid":"b4e2dc8c-98c7-4be5-a0a2-917ec54e5910","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2486862776},{"type":"v1","addr":"172.21.15.163:6817","nonce":2486862776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2486862776},{"type":"v1","addr":"172.21.15.163:6819","nonce":2486862776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2486862776},{"type":"v1","addr":"172.21.15.163:6823","nonce":2486862776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2486862776},{"type":"v1","addr":"172.21.15.163:6821","nonce":2486862776}]},"public_addr":"172.21.15.163:6817/2486862776","cluster_addr":"172.21.15.163:6819/2486862776","heartbeat_back_addr":"172.21.15.163:6823/2486862776","heartbeat_front_addr":"172.21.15.163:6821/2486862776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:56:27.613880+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:56:55.525968+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:57:22.785587+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:57:47.270057+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:58:12.791831+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:58:38.648113+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/1864158862":"2024-03-12T22:55:18.043211+0000","172.21.15.153:0/41949006":"2024-03-12T22:55:18.043211+0000","172.21.15.153:6801/4212724556":"2024-03-12T22:55:18.043211+0000","172.21.15.153:0/369153497":"2024-03-12T22:55:18.043211+0000","172.21.15.153:6800/4212724556":"2024-03-12T22:55:18.043211+0000","172.21.15.153:6801/1518383615":"2024-03-12T22:54:17.065945+0000","172.21.15.153:6800/1518383615":"2024-03-12T22:54:17.065945+0000","172.21.15.153:0/1974568910":"2024-03-12T22:54:17.065945+0000","172.21.15.153:0/2653190270":"2024-03-12T22:54:17.065945+0000","172.21.15.153:6800/2578439373":"2024-03-12T22:53:46.835889+0000","172.21.15.153:6801/2578439373":"2024-03-12T22:53:46.835889+0000","172.21.15.153:0/3587313628":"2024-03-12T22:53:46.835889+0000","172.21.15.153:0/3414319920":"2024-03-12T22:54:17.065945+0000","172.21.15.153:0/3612064784":"2024-03-12T22:53:46.835889+0000","172.21.15.153:0/3032487131":"2024-03-12T22:53:46.835889+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T22:59:00.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:00.024+0000 7f100eb5b700 1 -- 172.21.15.153:0/2806916709 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f0ff00692a0 msgr2=0x7f0ff006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:00.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:00.024+0000 7f100eb5b700 1 --2- 172.21.15.153:0/2806916709 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f0ff00692a0 0x7f0ff006b750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1000003400 tx=0x7f1000004d10 comp rx=0 tx=0).stop 2024-03-11T22:59:00.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:00.024+0000 7f100eb5b700 1 -- 172.21.15.153:0/2806916709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1008070be0 msgr2=0x7f10081162c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:00.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:00.024+0000 7f100eb5b700 1 --2- 172.21.15.153:0/2806916709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1008070be0 0x7f10081162c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc002a80 tx=0x7f0ffc0046b0 comp rx=0 tx=0).stop 2024-03-11T22:59:00.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:00.024+0000 7f100eb5b700 1 -- 172.21.15.153:0/2806916709 shutdown_connections 2024-03-11T22:59:00.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:00.024+0000 7f100eb5b700 1 --2- 172.21.15.153:0/2806916709 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f0ff00692a0 0x7f0ff006b750 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:00.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:00.024+0000 7f100eb5b700 1 --2- 172.21.15.153:0/2806916709 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f10080702d0 0x7f1008115d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:00.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:00.024+0000 7f100eb5b700 1 --2- 172.21.15.153:0/2806916709 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1008070be0 0x7f10081162c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:00.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:00.024+0000 7f100eb5b700 1 -- 172.21.15.153:0/2806916709 >> 172.21.15.153:0/2806916709 conn(0x7f10080f8ae0 msgr2=0x7f1008103b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:00.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:00.024+0000 7f100eb5b700 1 -- 172.21.15.153:0/2806916709 shutdown_connections 2024-03-11T22:59:00.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:00.024+0000 7f100eb5b700 1 -- 172.21.15.153:0/2806916709 wait complete. 2024-03-11T22:59:00.724 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:00 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/2806916709' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:59:00.737 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:00 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/2806916709' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:59:00.750 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-11T22:59:00.751 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T22:59:01.005 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:59:01.435 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:01 smithi153 ceph-mon[28962]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:01.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:01 smithi163 ceph-mon[35584]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:02.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.604+0000 7f6358958700 1 -- 172.21.15.153:0/1342754500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6354102f40 msgr2=0x7f6354103310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:02.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.604+0000 7f6358958700 1 --2- 172.21.15.153:0/1342754500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6354102f40 0x7f6354103310 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6344004940 tx=0x7f6344009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:02.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.604+0000 7f6358958700 1 -- 172.21.15.153:0/1342754500 shutdown_connections 2024-03-11T22:59:02.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.604+0000 7f6358958700 1 --2- 172.21.15.153:0/1342754500 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63541038e0 0x7f63540fcf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:02.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.604+0000 7f6358958700 1 --2- 172.21.15.153:0/1342754500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6354102f40 0x7f6354103310 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:02.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.604+0000 7f6358958700 1 -- 172.21.15.153:0/1342754500 >> 172.21.15.153:0/1342754500 conn(0x7f63540f8af0 msgr2=0x7f63540faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:02.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.604+0000 7f6358958700 1 -- 172.21.15.153:0/1342754500 shutdown_connections 2024-03-11T22:59:02.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.604+0000 7f6358958700 1 -- 172.21.15.153:0/1342754500 wait complete. 2024-03-11T22:59:02.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.604+0000 7f6358958700 1 Processor -- start 2024-03-11T22:59:02.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.604+0000 7f6358958700 1 -- start start 2024-03-11T22:59:02.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.604+0000 7f6358958700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6354102f40 0x7f6354113a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:02.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.604+0000 7f6358958700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63541038e0 0x7f6354113f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:02.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.604+0000 7f6358958700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63541145d0 con 0x7f6354102f40 2024-03-11T22:59:02.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.604+0000 7f6358958700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f635410d900 con 0x7f63541038e0 2024-03-11T22:59:02.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.605+0000 7f635259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6354102f40 0x7f6354113a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:02.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.605+0000 7f635259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6354102f40 0x7f6354113a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54710/0 (socket says 172.21.15.153:54710) 2024-03-11T22:59:02.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.605+0000 7f635259c700 1 -- 172.21.15.153:0/145382101 learned_addr learned my addr 172.21.15.153:0/145382101 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:59:02.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.605+0000 7f635259c700 1 -- 172.21.15.153:0/145382101 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63541038e0 msgr2=0x7f6354113f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:02.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.605+0000 7f635259c700 1 --2- 172.21.15.153:0/145382101 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63541038e0 0x7f6354113f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:02.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.605+0000 7f635259c700 1 -- 172.21.15.153:0/145382101 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6344005040 con 0x7f6354102f40 2024-03-11T22:59:02.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.605+0000 7f635259c700 1 --2- 172.21.15.153:0/145382101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6354102f40 0x7f6354113a40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f634400b3b0 tx=0x7f6344009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:02.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.605+0000 7f6342ffd700 1 -- 172.21.15.153:0/145382101 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f634400e040 con 0x7f6354102f40 2024-03-11T22:59:02.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.605+0000 7f6342ffd700 1 -- 172.21.15.153:0/145382101 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f634400bbf0 con 0x7f6354102f40 2024-03-11T22:59:02.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.605+0000 7f6342ffd700 1 -- 172.21.15.153:0/145382101 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f634401c630 con 0x7f6354102f40 2024-03-11T22:59:02.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.605+0000 7f6358958700 1 -- 172.21.15.153:0/145382101 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f635410db80 con 0x7f6354102f40 2024-03-11T22:59:02.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.605+0000 7f6358958700 1 -- 172.21.15.153:0/145382101 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f635410e0c0 con 0x7f6354102f40 2024-03-11T22:59:02.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.606+0000 7f6358958700 1 -- 172.21.15.153:0/145382101 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63540fc6f0 con 0x7f6354102f40 2024-03-11T22:59:02.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.607+0000 7f6342ffd700 1 -- 172.21.15.153:0/145382101 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f634400bd60 con 0x7f6354102f40 2024-03-11T22:59:02.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.607+0000 7f6342ffd700 1 --2- 172.21.15.153:0/145382101 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f633c0692a0 0x7f633c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:02.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.607+0000 7f6342ffd700 1 -- 172.21.15.153:0/145382101 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6344088240 con 0x7f6354102f40 2024-03-11T22:59:02.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.607+0000 7f6351d9b700 1 --2- 172.21.15.153:0/145382101 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f633c0692a0 0x7f633c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:02.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.607+0000 7f6351d9b700 1 --2- 172.21.15.153:0/145382101 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f633c0692a0 0x7f633c06b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f635410f270 tx=0x7f6348007d40 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:02.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.612+0000 7f6342ffd700 1 -- 172.21.15.153:0/145382101 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f6344058340 con 0x7f6354102f40 2024-03-11T22:59:02.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.800+0000 7f6358958700 1 -- 172.21.15.153:0/145382101 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f635404bed0 con 0x7f6354102f40 2024-03-11T22:59:02.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.801+0000 7f6342ffd700 1 -- 172.21.15.153:0/145382101 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11145 (secure 0 0 0) 0x7f6344057ed0 con 0x7f6354102f40 2024-03-11T22:59:02.805 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:59:02.805 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":41,"fsid":"fa913b96-dff9-11ee-95c8-87774f69a715","created":"2024-03-11T22:53:14.269171+0000","modified":"2024-03-11T22:58:41.287239+0000","last_up_change":"2024-03-11T22:58:40.147211+0000","last_in_change":"2024-03-11T22:58:21.268596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T22:57:26.368808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"542731eb-3fb3-450b-93d6-c6a8a9df1659","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1406194697},{"type":"v1","addr":"172.21.15.153:6803","nonce":1406194697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1406194697},{"type":"v1","addr":"172.21.15.153:6805","nonce":1406194697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1406194697},{"type":"v1","addr":"172.21.15.153:6809","nonce":1406194697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1406194697},{"type":"v1","addr":"172.21.15.153:6807","nonce":1406194697}]},"public_addr":"172.21.15.153:6803/1406194697","cluster_addr":"172.21.15.153:6805/1406194697","heartbeat_back_addr":"172.21.15.153:6809/1406194697","heartbeat_front_addr":"172.21.15.153:6807/1406194697","state":["exists","up"]},{"osd":1,"uuid":"7be06d22-d83a-4b39-ace8-3d800bbd1aee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2718652372},{"type":"v1","addr":"172.21.15.153:6811","nonce":2718652372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2718652372},{"type":"v1","addr":"172.21.15.153:6813","nonce":2718652372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2718652372},{"type":"v1","addr":"172.21.15.153:6817","nonce":2718652372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2718652372},{"type":"v1","addr":"172.21.15.153:6815","nonce":2718652372}]},"public_addr":"172.21.15.153:6811/2718652372","cluster_addr":"172.21.15.153:6813/2718652372","heartbeat_back_addr":"172.21.15.153:6817/2718652372","heartbeat_front_addr":"172.21.15.153:6815/2718652372","state":["exists","up"]},{"osd":2,"uuid":"b4863162-3a3d-45ae-9bc0-c8b069ff063a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":4016153774},{"type":"v1","addr":"172.21.15.153:6819","nonce":4016153774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":4016153774},{"type":"v1","addr":"172.21.15.153:6821","nonce":4016153774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4016153774},{"type":"v1","addr":"172.21.15.153:6825","nonce":4016153774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":4016153774},{"type":"v1","addr":"172.21.15.153:6823","nonce":4016153774}]},"public_addr":"172.21.15.153:6819/4016153774","cluster_addr":"172.21.15.153:6821/4016153774","heartbeat_back_addr":"172.21.15.153:6825/4016153774","heartbeat_front_addr":"172.21.15.153:6823/4016153774","state":["exists","up"]},{"osd":3,"uuid":"92a62b43-e245-4d71-b4ca-89a7d518af50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":3768791983},{"type":"v1","addr":"172.21.15.163:6801","nonce":3768791983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":3768791983},{"type":"v1","addr":"172.21.15.163:6803","nonce":3768791983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":3768791983},{"type":"v1","addr":"172.21.15.163:6807","nonce":3768791983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":3768791983},{"type":"v1","addr":"172.21.15.163:6805","nonce":3768791983}]},"public_addr":"172.21.15.163:6801/3768791983","cluster_addr":"172.21.15.163:6803/3768791983","heartbeat_back_addr":"172.21.15.163:6807/3768791983","heartbeat_front_addr":"172.21.15.163:6805/3768791983","state":["exists","up"]},{"osd":4,"uuid":"b9caf0a9-46bf-44bc-a2d0-b5f0e1b3eadf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":4243752277},{"type":"v1","addr":"172.21.15.163:6809","nonce":4243752277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":4243752277},{"type":"v1","addr":"172.21.15.163:6811","nonce":4243752277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":4243752277},{"type":"v1","addr":"172.21.15.163:6815","nonce":4243752277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":4243752277},{"type":"v1","addr":"172.21.15.163:6813","nonce":4243752277}]},"public_addr":"172.21.15.163:6809/4243752277","cluster_addr":"172.21.15.163:6811/4243752277","heartbeat_back_addr":"172.21.15.163:6815/4243752277","heartbeat_front_addr":"172.21.15.163:6813/4243752277","state":["exists","up"]},{"osd":5,"uuid":"b4e2dc8c-98c7-4be5-a0a2-917ec54e5910","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2486862776},{"type":"v1","addr":"172.21.15.163:6817","nonce":2486862776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2486862776},{"type":"v1","addr":"172.21.15.163:6819","nonce":2486862776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2486862776},{"type":"v1","addr":"172.21.15.163:6823","nonce":2486862776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2486862776},{"type":"v1","addr":"172.21.15.163:6821","nonce":2486862776}]},"public_addr":"172.21.15.163:6817/2486862776","cluster_addr":"172.21.15.163:6819/2486862776","heartbeat_back_addr":"172.21.15.163:6823/2486862776","heartbeat_front_addr":"172.21.15.163:6821/2486862776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:56:27.613880+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:56:55.525968+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:57:22.785587+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:57:47.270057+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:58:12.791831+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:58:38.648113+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/1864158862":"2024-03-12T22:55:18.043211+0000","172.21.15.153:0/41949006":"2024-03-12T22:55:18.043211+0000","172.21.15.153:6801/4212724556":"2024-03-12T22:55:18.043211+0000","172.21.15.153:0/369153497":"2024-03-12T22:55:18.043211+0000","172.21.15.153:6800/4212724556":"2024-03-12T22:55:18.043211+0000","172.21.15.153:6801/1518383615":"2024-03-12T22:54:17.065945+0000","172.21.15.153:6800/1518383615":"2024-03-12T22:54:17.065945+0000","172.21.15.153:0/1974568910":"2024-03-12T22:54:17.065945+0000","172.21.15.153:0/2653190270":"2024-03-12T22:54:17.065945+0000","172.21.15.153:6800/2578439373":"2024-03-12T22:53:46.835889+0000","172.21.15.153:6801/2578439373":"2024-03-12T22:53:46.835889+0000","172.21.15.153:0/3587313628":"2024-03-12T22:53:46.835889+0000","172.21.15.153:0/3414319920":"2024-03-12T22:54:17.065945+0000","172.21.15.153:0/3612064784":"2024-03-12T22:53:46.835889+0000","172.21.15.153:0/3032487131":"2024-03-12T22:53:46.835889+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T22:59:02.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.803+0000 7f6358958700 1 -- 172.21.15.153:0/145382101 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f633c0692a0 msgr2=0x7f633c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:02.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.803+0000 7f6358958700 1 --2- 172.21.15.153:0/145382101 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f633c0692a0 0x7f633c06b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f635410f270 tx=0x7f6348007d40 comp rx=0 tx=0).stop 2024-03-11T22:59:02.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.803+0000 7f6358958700 1 -- 172.21.15.153:0/145382101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6354102f40 msgr2=0x7f6354113a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:02.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.803+0000 7f6358958700 1 --2- 172.21.15.153:0/145382101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6354102f40 0x7f6354113a40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f634400b3b0 tx=0x7f6344009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:02.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.803+0000 7f6358958700 1 -- 172.21.15.153:0/145382101 shutdown_connections 2024-03-11T22:59:02.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.803+0000 7f6358958700 1 --2- 172.21.15.153:0/145382101 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6354102f40 0x7f6354113a40 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:02.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.803+0000 7f6358958700 1 --2- 172.21.15.153:0/145382101 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f633c0692a0 0x7f633c06b750 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:02.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.803+0000 7f6358958700 1 --2- 172.21.15.153:0/145382101 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63541038e0 0x7f6354113f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:02.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.803+0000 7f6358958700 1 -- 172.21.15.153:0/145382101 >> 172.21.15.153:0/145382101 conn(0x7f63540f8af0 msgr2=0x7f63540fa560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:02.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.803+0000 7f6358958700 1 -- 172.21.15.153:0/145382101 shutdown_connections 2024-03-11T22:59:02.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:02.803+0000 7f6358958700 1 -- 172.21.15.153:0/145382101 wait complete. 2024-03-11T22:59:03.507 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-11T22:59:03.507 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-11T22:59:03.508 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-11T22:59:03.508 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-11T22:59:03.508 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-11T22:59:03.509 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-11T22:59:03.541 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:03 smithi153 ceph-mon[28962]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:03.541 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:03 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/145382101' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:59:03.541 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:03 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:59:03.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:03 smithi163 ceph-mon[35584]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:03.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:03 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/145382101' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:59:03.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:03 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:59:03.820 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:59:03.826 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:59:05.723 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:05 smithi153 ceph-mon[28962]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:05.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:05 smithi163 ceph-mon[35584]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:05.808 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:59:07.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.116+0000 7fdb192ba700 1 -- 172.21.15.153:0/3766791277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb14101e50 msgr2=0x7fdb14102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:07.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.116+0000 7fdb192ba700 1 --2- 172.21.15.153:0/3766791277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb14101e50 0x7fdb14102220 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fdb04004940 tx=0x7fdb04009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:07.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.116+0000 7fdb192ba700 1 -- 172.21.15.153:0/3766791277 shutdown_connections 2024-03-11T22:59:07.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.116+0000 7fdb192ba700 1 --2- 172.21.15.153:0/3766791277 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb141027f0 0x7fdb140fbe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:07.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.116+0000 7fdb192ba700 1 --2- 172.21.15.153:0/3766791277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb14101e50 0x7fdb14102220 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:07.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.116+0000 7fdb192ba700 1 -- 172.21.15.153:0/3766791277 >> 172.21.15.153:0/3766791277 conn(0x7fdb140f79e0 msgr2=0x7fdb140f9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:07.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.116+0000 7fdb192ba700 1 -- 172.21.15.153:0/3766791277 shutdown_connections 2024-03-11T22:59:07.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.116+0000 7fdb192ba700 1 -- 172.21.15.153:0/3766791277 wait complete. 2024-03-11T22:59:07.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.117+0000 7fdb192ba700 1 Processor -- start 2024-03-11T22:59:07.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.117+0000 7fdb192ba700 1 -- start start 2024-03-11T22:59:07.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.117+0000 7fdb192ba700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb14101e50 0x7fdb1410eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:07.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.117+0000 7fdb192ba700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb141027f0 0x7fdb1410f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:07.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.117+0000 7fdb192ba700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1410f5f0 con 0x7fdb14101e50 2024-03-11T22:59:07.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.117+0000 7fdb192ba700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1410f730 con 0x7fdb141027f0 2024-03-11T22:59:07.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.118+0000 7fdb12ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb14101e50 0x7fdb1410eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:07.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.118+0000 7fdb12ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb14101e50 0x7fdb1410eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54734/0 (socket says 172.21.15.153:54734) 2024-03-11T22:59:07.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.118+0000 7fdb12ffd700 1 -- 172.21.15.153:0/3074685034 learned_addr learned my addr 172.21.15.153:0/3074685034 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:59:07.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.118+0000 7fdb12ffd700 1 -- 172.21.15.153:0/3074685034 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb141027f0 msgr2=0x7fdb1410f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:07.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.118+0000 7fdb127fc700 1 --2- 172.21.15.153:0/3074685034 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb141027f0 0x7fdb1410f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:07.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.118+0000 7fdb12ffd700 1 --2- 172.21.15.153:0/3074685034 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb141027f0 0x7fdb1410f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:07.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.118+0000 7fdb12ffd700 1 -- 172.21.15.153:0/3074685034 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb04005040 con 0x7fdb14101e50 2024-03-11T22:59:07.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.118+0000 7fdb12ffd700 1 --2- 172.21.15.153:0/3074685034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb14101e50 0x7fdb1410eb20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fdb04009fd0 tx=0x7fdb04009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:07.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.118+0000 7fdafb7fe700 1 -- 172.21.15.153:0/3074685034 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb0400e040 con 0x7fdb14101e50 2024-03-11T22:59:07.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.118+0000 7fdb192ba700 1 -- 172.21.15.153:0/3074685034 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb14117a30 con 0x7fdb14101e50 2024-03-11T22:59:07.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.118+0000 7fdafb7fe700 1 -- 172.21.15.153:0/3074685034 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdb0400bbf0 con 0x7fdb14101e50 2024-03-11T22:59:07.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.118+0000 7fdafb7fe700 1 -- 172.21.15.153:0/3074685034 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb0401c3f0 con 0x7fdb14101e50 2024-03-11T22:59:07.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.118+0000 7fdb192ba700 1 -- 172.21.15.153:0/3074685034 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb14117f40 con 0x7fdb14101e50 2024-03-11T22:59:07.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.119+0000 7fdb192ba700 1 -- 172.21.15.153:0/3074685034 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdb1405e690 con 0x7fdb14101e50 2024-03-11T22:59:07.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.120+0000 7fdafb7fe700 1 -- 172.21.15.153:0/3074685034 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fdb0400bd60 con 0x7fdb14101e50 2024-03-11T22:59:07.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.121+0000 7fdafb7fe700 1 --2- 172.21.15.153:0/3074685034 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fdafc0692b0 0x7fdafc06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:07.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.121+0000 7fdafb7fe700 1 -- 172.21.15.153:0/3074685034 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fdb04088c10 con 0x7fdb14101e50 2024-03-11T22:59:07.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.121+0000 7fdb127fc700 1 --2- 172.21.15.153:0/3074685034 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fdafc0692b0 0x7fdafc06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:07.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.121+0000 7fdafb7fe700 1 --2- 172.21.15.153:0/3074685034 >> [v2:172.21.15.163:6816/2486862776,v1:172.21.15.163:6817/2486862776] conn(0x7fdafc06ef80 0x7fdafc071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:07.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.121+0000 7fdafb7fe700 1 -- 172.21.15.153:0/3074685034 --> [v2:172.21.15.163:6816/2486862776,v1:172.21.15.163:6817/2486862776] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdafc071a40 con 0x7fdafc06ef80 2024-03-11T22:59:07.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.121+0000 7fdafb7fe700 1 -- 172.21.15.153:0/3074685034 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdb04088ff0 con 0x7fdb14101e50 2024-03-11T22:59:07.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.121+0000 7fdb127fc700 1 --2- 172.21.15.153:0/3074685034 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fdafc0692b0 0x7fdafc06b760 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdb14066a10 tx=0x7fdb08009040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:07.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.121+0000 7fdb137fe700 1 --2- 172.21.15.153:0/3074685034 >> [v2:172.21.15.163:6816/2486862776,v1:172.21.15.163:6817/2486862776] conn(0x7fdafc06ef80 0x7fdafc071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:07.126 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.122+0000 7fdb137fe700 1 --2- 172.21.15.153:0/3074685034 >> [v2:172.21.15.163:6816/2486862776,v1:172.21.15.163:6817/2486862776] conn(0x7fdafc06ef80 0x7fdafc071390 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:07.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.122+0000 7fdafb7fe700 1 -- 172.21.15.153:0/3074685034 <== osd.5 v2:172.21.15.163:6816/2486862776 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fdafc071a40 con 0x7fdafc06ef80 2024-03-11T22:59:07.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.137+0000 7fdb192ba700 1 -- 172.21.15.153:0/3074685034 --> [v2:172.21.15.163:6816/2486862776,v1:172.21.15.163:6817/2486862776] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdb1405eb10 con 0x7fdafc06ef80 2024-03-11T22:59:07.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.137+0000 7fdafb7fe700 1 -- 172.21.15.153:0/3074685034 <== osd.5 v2:172.21.15.163:6816/2486862776 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fdb1405eb10 con 0x7fdafc06ef80 2024-03-11T22:59:07.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.138+0000 7fdb192ba700 1 -- 172.21.15.153:0/3074685034 >> [v2:172.21.15.163:6816/2486862776,v1:172.21.15.163:6817/2486862776] conn(0x7fdafc06ef80 msgr2=0x7fdafc071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:07.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.138+0000 7fdb192ba700 1 --2- 172.21.15.153:0/3074685034 >> [v2:172.21.15.163:6816/2486862776,v1:172.21.15.163:6817/2486862776] conn(0x7fdafc06ef80 0x7fdafc071390 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:07.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.138+0000 7fdb192ba700 1 -- 172.21.15.153:0/3074685034 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fdafc0692b0 msgr2=0x7fdafc06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:07.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.138+0000 7fdb192ba700 1 --2- 172.21.15.153:0/3074685034 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fdafc0692b0 0x7fdafc06b760 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdb14066a10 tx=0x7fdb08009040 comp rx=0 tx=0).stop 2024-03-11T22:59:07.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.138+0000 7fdb192ba700 1 -- 172.21.15.153:0/3074685034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb14101e50 msgr2=0x7fdb1410eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:07.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.138+0000 7fdb192ba700 1 --2- 172.21.15.153:0/3074685034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb14101e50 0x7fdb1410eb20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fdb04009fd0 tx=0x7fdb04009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:07.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.138+0000 7fdb192ba700 1 -- 172.21.15.153:0/3074685034 shutdown_connections 2024-03-11T22:59:07.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.138+0000 7fdb192ba700 1 --2- 172.21.15.153:0/3074685034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb14101e50 0x7fdb1410eb20 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:07.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.138+0000 7fdb192ba700 1 --2- 172.21.15.153:0/3074685034 >> [v2:172.21.15.163:6816/2486862776,v1:172.21.15.163:6817/2486862776] conn(0x7fdafc06ef80 0x7fdafc071390 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:07.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.138+0000 7fdb192ba700 1 --2- 172.21.15.153:0/3074685034 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fdafc0692b0 0x7fdafc06b760 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:07.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.138+0000 7fdb192ba700 1 --2- 172.21.15.153:0/3074685034 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdb141027f0 0x7fdb1410f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:07.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.138+0000 7fdb192ba700 1 -- 172.21.15.153:0/3074685034 >> 172.21.15.153:0/3074685034 conn(0x7fdb140f79e0 msgr2=0x7fdb140f9450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:07.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.138+0000 7fdb192ba700 1 -- 172.21.15.153:0/3074685034 shutdown_connections 2024-03-11T22:59:07.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.138+0000 7fdb192ba700 1 -- 172.21.15.153:0/3074685034 wait complete. 2024-03-11T22:59:07.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:07 smithi153 ceph-mon[28962]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:07.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:07 smithi163 ceph-mon[35584]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:07.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.759+0000 7fd80f54f700 1 -- 172.21.15.153:0/3896464258 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd808102800 msgr2=0x7fd8080fbe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:07.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.759+0000 7fd80f54f700 1 --2- 172.21.15.153:0/3896464258 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd808102800 0x7fd8080fbe70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8004940 tx=0x7fd7f8009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:07.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.759+0000 7fd80f54f700 1 -- 172.21.15.153:0/3896464258 shutdown_connections 2024-03-11T22:59:07.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.759+0000 7fd80f54f700 1 --2- 172.21.15.153:0/3896464258 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd808102800 0x7fd8080fbe70 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:07.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.759+0000 7fd80f54f700 1 --2- 172.21.15.153:0/3896464258 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd808101e60 0x7fd808102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:07.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.759+0000 7fd80f54f700 1 -- 172.21.15.153:0/3896464258 >> 172.21.15.153:0/3896464258 conn(0x7fd8080f7a10 msgr2=0x7fd8080f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:07.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.759+0000 7fd80f54f700 1 -- 172.21.15.153:0/3896464258 shutdown_connections 2024-03-11T22:59:07.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.759+0000 7fd80f54f700 1 -- 172.21.15.153:0/3896464258 wait complete. 2024-03-11T22:59:07.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.759+0000 7fd80f54f700 1 Processor -- start 2024-03-11T22:59:07.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.760+0000 7fd80f54f700 1 -- start start 2024-03-11T22:59:07.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.760+0000 7fd80f54f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd808101e60 0x7fd808113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:07.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.760+0000 7fd80f54f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd808102800 0x7fd808114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:07.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.760+0000 7fd80f54f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd808114700 con 0x7fd808102800 2024-03-11T22:59:07.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.760+0000 7fd80f54f700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd80810d9a0 con 0x7fd808101e60 2024-03-11T22:59:07.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.760+0000 7fd80caea700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd808102800 0x7fd808114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:07.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.760+0000 7fd80caea700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd808102800 0x7fd808114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54744/0 (socket says 172.21.15.153:54744) 2024-03-11T22:59:07.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.760+0000 7fd80caea700 1 -- 172.21.15.153:0/3813635760 learned_addr learned my addr 172.21.15.153:0/3813635760 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:59:07.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.760+0000 7fd80d2eb700 1 --2- 172.21.15.153:0/3813635760 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd808101e60 0x7fd808113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:07.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.760+0000 7fd80caea700 1 -- 172.21.15.153:0/3813635760 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd808101e60 msgr2=0x7fd808113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:07.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.760+0000 7fd80caea700 1 --2- 172.21.15.153:0/3813635760 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd808101e60 0x7fd808113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:07.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.760+0000 7fd80caea700 1 -- 172.21.15.153:0/3813635760 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7f8005040 con 0x7fd808102800 2024-03-11T22:59:07.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.760+0000 7fd80caea700 1 --2- 172.21.15.153:0/3813635760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd808102800 0x7fd808114020 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd7f80090d0 tx=0x7fd7f8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:07.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.760+0000 7fd7fdffb700 1 -- 172.21.15.153:0/3813635760 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7f800e040 con 0x7fd808102800 2024-03-11T22:59:07.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.760+0000 7fd80f54f700 1 -- 172.21.15.153:0/3813635760 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd80810dc20 con 0x7fd808102800 2024-03-11T22:59:07.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.760+0000 7fd7fdffb700 1 -- 172.21.15.153:0/3813635760 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd7f800bbf0 con 0x7fd808102800 2024-03-11T22:59:07.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.761+0000 7fd7fdffb700 1 -- 172.21.15.153:0/3813635760 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7f801c630 con 0x7fd808102800 2024-03-11T22:59:07.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.761+0000 7fd80f54f700 1 -- 172.21.15.153:0/3813635760 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd80810e160 con 0x7fd808102800 2024-03-11T22:59:07.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.761+0000 7fd80f54f700 1 -- 172.21.15.153:0/3813635760 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd80810e740 con 0x7fd808102800 2024-03-11T22:59:07.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.762+0000 7fd7fdffb700 1 -- 172.21.15.153:0/3813635760 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fd7f800bd60 con 0x7fd808102800 2024-03-11T22:59:07.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.762+0000 7fd7fdffb700 1 --2- 172.21.15.153:0/3813635760 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fd7f40692a0 0x7fd7f406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:07.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.763+0000 7fd7fdffb700 1 -- 172.21.15.153:0/3813635760 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd7f8088c30 con 0x7fd808102800 2024-03-11T22:59:07.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.763+0000 7fd80d2eb700 1 --2- 172.21.15.153:0/3813635760 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fd7f40692a0 0x7fd7f406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:07.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.763+0000 7fd7fdffb700 1 --2- 172.21.15.153:0/3813635760 >> [v2:172.21.15.153:6802/1406194697,v1:172.21.15.153:6803/1406194697] conn(0x7fd7f406ef70 0x7fd7f4071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:07.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.763+0000 7fd7fdffb700 1 -- 172.21.15.153:0/3813635760 --> [v2:172.21.15.153:6802/1406194697,v1:172.21.15.153:6803/1406194697] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd7f4071a30 con 0x7fd7f406ef70 2024-03-11T22:59:07.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.763+0000 7fd7fdffb700 1 -- 172.21.15.153:0/3813635760 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7f8089010 con 0x7fd808102800 2024-03-11T22:59:07.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.763+0000 7fd80daec700 1 --2- 172.21.15.153:0/3813635760 >> [v2:172.21.15.153:6802/1406194697,v1:172.21.15.153:6803/1406194697] conn(0x7fd7f406ef70 0x7fd7f4071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:07.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.763+0000 7fd80d2eb700 1 --2- 172.21.15.153:0/3813635760 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fd7f40692a0 0x7fd7f406b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd80400a900 tx=0x7fd804008040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:07.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.763+0000 7fd80daec700 1 --2- 172.21.15.153:0/3813635760 >> [v2:172.21.15.153:6802/1406194697,v1:172.21.15.153:6803/1406194697] conn(0x7fd7f406ef70 0x7fd7f4071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:07.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.764+0000 7fd7fdffb700 1 -- 172.21.15.153:0/3813635760 <== osd.0 v2:172.21.15.153:6802/1406194697 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fd7f4071a30 con 0x7fd7f406ef70 2024-03-11T22:59:07.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.779+0000 7fd80f54f700 1 -- 172.21.15.153:0/3813635760 --> [v2:172.21.15.153:6802/1406194697,v1:172.21.15.153:6803/1406194697] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd8080669a0 con 0x7fd7f406ef70 2024-03-11T22:59:07.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.779+0000 7fd7fdffb700 1 -- 172.21.15.153:0/3813635760 <== osd.0 v2:172.21.15.153:6802/1406194697 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd8080669a0 con 0x7fd7f406ef70 2024-03-11T22:59:07.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.779+0000 7fd80f54f700 1 -- 172.21.15.153:0/3813635760 >> [v2:172.21.15.153:6802/1406194697,v1:172.21.15.153:6803/1406194697] conn(0x7fd7f406ef70 msgr2=0x7fd7f4071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:07.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.779+0000 7fd80f54f700 1 --2- 172.21.15.153:0/3813635760 >> [v2:172.21.15.153:6802/1406194697,v1:172.21.15.153:6803/1406194697] conn(0x7fd7f406ef70 0x7fd7f4071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:07.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.779+0000 7fd80f54f700 1 -- 172.21.15.153:0/3813635760 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fd7f40692a0 msgr2=0x7fd7f406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:07.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.779+0000 7fd80f54f700 1 --2- 172.21.15.153:0/3813635760 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fd7f40692a0 0x7fd7f406b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd80400a900 tx=0x7fd804008040 comp rx=0 tx=0).stop 2024-03-11T22:59:07.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.779+0000 7fd80f54f700 1 -- 172.21.15.153:0/3813635760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd808102800 msgr2=0x7fd808114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:07.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.779+0000 7fd80f54f700 1 --2- 172.21.15.153:0/3813635760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd808102800 0x7fd808114020 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd7f80090d0 tx=0x7fd7f8009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:07.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.780+0000 7fd80f54f700 1 -- 172.21.15.153:0/3813635760 shutdown_connections 2024-03-11T22:59:07.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.780+0000 7fd80f54f700 1 --2- 172.21.15.153:0/3813635760 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fd7f40692a0 0x7fd7f406b750 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:07.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.780+0000 7fd80f54f700 1 --2- 172.21.15.153:0/3813635760 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd808101e60 0x7fd808113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:07.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.780+0000 7fd80f54f700 1 --2- 172.21.15.153:0/3813635760 >> [v2:172.21.15.153:6802/1406194697,v1:172.21.15.153:6803/1406194697] conn(0x7fd7f406ef70 0x7fd7f4071380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:07.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.780+0000 7fd80f54f700 1 --2- 172.21.15.153:0/3813635760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd808102800 0x7fd808114020 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:07.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.780+0000 7fd80f54f700 1 -- 172.21.15.153:0/3813635760 >> 172.21.15.153:0/3813635760 conn(0x7fd8080f7a10 msgr2=0x7fd808065f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:07.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.780+0000 7fd80f54f700 1 -- 172.21.15.153:0/3813635760 shutdown_connections 2024-03-11T22:59:07.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:07.780+0000 7fd80f54f700 1 -- 172.21.15.153:0/3813635760 wait complete. 2024-03-11T22:59:08.492 INFO:teuthology.orchestra.run.smithi153.stdout:38654705698 2024-03-11T22:59:08.493 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-11T22:59:08.526 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:59:08.927 INFO:teuthology.orchestra.run.smithi153.stdout:171798691848 2024-03-11T22:59:08.927 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-11T22:59:08.939 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:59:08.941 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:59:09.483 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:09 smithi153 ceph-mon[28962]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:09.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:09 smithi163 ceph-mon[35584]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:11.113 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:59:11.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:11 smithi153 ceph-mon[28962]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:11.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:11 smithi163 ceph-mon[35584]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:13.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.932+0000 7faa11845700 1 -- 172.21.15.153:0/250946750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa0c0fdf50 msgr2=0x7faa0c0fe320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:13.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.932+0000 7faa11845700 1 --2- 172.21.15.153:0/250946750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa0c0fdf50 0x7faa0c0fe320 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc004940 tx=0x7fa9fc009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:13.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.932+0000 7faa11845700 1 -- 172.21.15.153:0/250946750 shutdown_connections 2024-03-11T22:59:13.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.932+0000 7faa11845700 1 --2- 172.21.15.153:0/250946750 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa0c0fe8f0 0x7faa0c106fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:13.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.932+0000 7faa11845700 1 --2- 172.21.15.153:0/250946750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa0c0fdf50 0x7faa0c0fe320 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:13.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.932+0000 7faa11845700 1 -- 172.21.15.153:0/250946750 >> 172.21.15.153:0/250946750 conn(0x7faa0c0f7a10 msgr2=0x7faa0c0f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:13.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.932+0000 7faa11845700 1 -- 172.21.15.153:0/250946750 shutdown_connections 2024-03-11T22:59:13.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.932+0000 7faa11845700 1 -- 172.21.15.153:0/250946750 wait complete. 2024-03-11T22:59:13.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.933+0000 7faa11845700 1 Processor -- start 2024-03-11T22:59:13.115 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.933+0000 7faa11845700 1 -- start start 2024-03-11T22:59:13.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.933+0000 7faa11845700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa0c0fdf50 0x7faa0c113a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:13.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.933+0000 7faa11845700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa0c0fe8f0 0x7faa0c113f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:13.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.933+0000 7faa11845700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa0c114660 con 0x7faa0c0fdf50 2024-03-11T22:59:13.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.933+0000 7faa11845700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa0c10d900 con 0x7faa0c0fe8f0 2024-03-11T22:59:13.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.933+0000 7faa0affd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa0c0fdf50 0x7faa0c113a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:13.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.933+0000 7faa0affd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa0c0fdf50 0x7faa0c113a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38246/0 (socket says 172.21.15.153:38246) 2024-03-11T22:59:13.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.933+0000 7faa0affd700 1 -- 172.21.15.153:0/187042731 learned_addr learned my addr 172.21.15.153:0/187042731 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:59:13.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.933+0000 7faa0a7fc700 1 --2- 172.21.15.153:0/187042731 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa0c0fe8f0 0x7faa0c113f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:13.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.933+0000 7faa0affd700 1 -- 172.21.15.153:0/187042731 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa0c0fe8f0 msgr2=0x7faa0c113f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:13.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.933+0000 7faa0affd700 1 --2- 172.21.15.153:0/187042731 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa0c0fe8f0 0x7faa0c113f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:13.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.933+0000 7faa0affd700 1 -- 172.21.15.153:0/187042731 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9fc005040 con 0x7faa0c0fdf50 2024-03-11T22:59:13.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.933+0000 7faa0affd700 1 --2- 172.21.15.153:0/187042731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa0c0fdf50 0x7faa0c113a40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc004890 tx=0x7fa9fc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:13.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.934+0000 7fa9f37fe700 1 -- 172.21.15.153:0/187042731 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9fc00e040 con 0x7faa0c0fdf50 2024-03-11T22:59:13.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.934+0000 7faa11845700 1 -- 172.21.15.153:0/187042731 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa0c10db80 con 0x7faa0c0fdf50 2024-03-11T22:59:13.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.934+0000 7fa9f37fe700 1 -- 172.21.15.153:0/187042731 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa9fc004290 con 0x7faa0c0fdf50 2024-03-11T22:59:13.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.934+0000 7fa9f37fe700 1 -- 172.21.15.153:0/187042731 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9fc01c630 con 0x7faa0c0fdf50 2024-03-11T22:59:13.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.934+0000 7faa11845700 1 -- 172.21.15.153:0/187042731 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa0c10e0c0 con 0x7faa0c0fdf50 2024-03-11T22:59:13.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.934+0000 7faa11845700 1 -- 172.21.15.153:0/187042731 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7faa0c10e560 con 0x7faa0c0fdf50 2024-03-11T22:59:13.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.935+0000 7fa9f37fe700 1 -- 172.21.15.153:0/187042731 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fa9fc00bbf0 con 0x7faa0c0fdf50 2024-03-11T22:59:13.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.935+0000 7fa9f37fe700 1 --2- 172.21.15.153:0/187042731 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fa9f40692a0 0x7fa9f406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:13.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.936+0000 7fa9f37fe700 1 -- 172.21.15.153:0/187042731 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa9fc088d40 con 0x7faa0c0fdf50 2024-03-11T22:59:13.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.936+0000 7faa0a7fc700 1 --2- 172.21.15.153:0/187042731 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fa9f40692a0 0x7fa9f406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:13.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.936+0000 7fa9f37fe700 1 --2- 172.21.15.153:0/187042731 >> [v2:172.21.15.153:6818/4016153774,v1:172.21.15.153:6819/4016153774] conn(0x7fa9f406ef70 0x7fa9f4071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:13.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.936+0000 7fa9f37fe700 1 -- 172.21.15.153:0/187042731 --> [v2:172.21.15.153:6818/4016153774,v1:172.21.15.153:6819/4016153774] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa9f4071a30 con 0x7fa9f406ef70 2024-03-11T22:59:13.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.936+0000 7fa9f37fe700 1 -- 172.21.15.153:0/187042731 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa9fc089120 con 0x7faa0c0fdf50 2024-03-11T22:59:13.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.936+0000 7faa0b7fe700 1 --2- 172.21.15.153:0/187042731 >> [v2:172.21.15.153:6818/4016153774,v1:172.21.15.153:6819/4016153774] conn(0x7fa9f406ef70 0x7fa9f4071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:13.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.936+0000 7faa0a7fc700 1 --2- 172.21.15.153:0/187042731 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fa9f40692a0 0x7fa9f406b750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7faa00006fd0 tx=0x7faa00008040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:13.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.936+0000 7faa0b7fe700 1 --2- 172.21.15.153:0/187042731 >> [v2:172.21.15.153:6818/4016153774,v1:172.21.15.153:6819/4016153774] conn(0x7fa9f406ef70 0x7fa9f4071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:13.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.937+0000 7fa9f37fe700 1 -- 172.21.15.153:0/187042731 <== osd.2 v2:172.21.15.153:6818/4016153774 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fa9f4071a30 con 0x7fa9f406ef70 2024-03-11T22:59:13.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.951+0000 7faa11845700 1 -- 172.21.15.153:0/187042731 --> [v2:172.21.15.153:6818/4016153774,v1:172.21.15.153:6819/4016153774] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7faa0c10e720 con 0x7fa9f406ef70 2024-03-11T22:59:13.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.952+0000 7fa9f37fe700 1 -- 172.21.15.153:0/187042731 <== osd.2 v2:172.21.15.153:6818/4016153774 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7faa0c10e720 con 0x7fa9f406ef70 2024-03-11T22:59:13.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.952+0000 7faa11845700 1 -- 172.21.15.153:0/187042731 >> [v2:172.21.15.153:6818/4016153774,v1:172.21.15.153:6819/4016153774] conn(0x7fa9f406ef70 msgr2=0x7fa9f4071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:13.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.952+0000 7faa11845700 1 --2- 172.21.15.153:0/187042731 >> [v2:172.21.15.153:6818/4016153774,v1:172.21.15.153:6819/4016153774] conn(0x7fa9f406ef70 0x7fa9f4071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:13.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.952+0000 7faa11845700 1 -- 172.21.15.153:0/187042731 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fa9f40692a0 msgr2=0x7fa9f406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:13.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.952+0000 7faa11845700 1 --2- 172.21.15.153:0/187042731 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fa9f40692a0 0x7fa9f406b750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7faa00006fd0 tx=0x7faa00008040 comp rx=0 tx=0).stop 2024-03-11T22:59:13.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.952+0000 7faa11845700 1 -- 172.21.15.153:0/187042731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa0c0fdf50 msgr2=0x7faa0c113a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:13.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.952+0000 7faa11845700 1 --2- 172.21.15.153:0/187042731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa0c0fdf50 0x7faa0c113a40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc004890 tx=0x7fa9fc009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:13.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.952+0000 7faa11845700 1 -- 172.21.15.153:0/187042731 shutdown_connections 2024-03-11T22:59:13.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.952+0000 7faa11845700 1 --2- 172.21.15.153:0/187042731 >> [v2:172.21.15.153:6818/4016153774,v1:172.21.15.153:6819/4016153774] conn(0x7fa9f406ef70 0x7fa9f4071380 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:13.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.952+0000 7faa11845700 1 --2- 172.21.15.153:0/187042731 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa0c0fdf50 0x7faa0c113a40 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:13.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.952+0000 7faa11845700 1 --2- 172.21.15.153:0/187042731 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fa9f40692a0 0x7fa9f406b750 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:13.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.952+0000 7faa11845700 1 --2- 172.21.15.153:0/187042731 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa0c0fe8f0 0x7faa0c113f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:13.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.952+0000 7faa11845700 1 -- 172.21.15.153:0/187042731 >> 172.21.15.153:0/187042731 conn(0x7faa0c0f7a10 msgr2=0x7faa0c1017a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:13.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.952+0000 7faa11845700 1 -- 172.21.15.153:0/187042731 shutdown_connections 2024-03-11T22:59:13.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:12.952+0000 7faa11845700 1 -- 172.21.15.153:0/187042731 wait complete. 2024-03-11T22:59:13.483 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:13 smithi153 ceph-mon[28962]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:13.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.577+0000 7fa257819700 1 -- 172.21.15.153:0/3907732015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2501038f0 msgr2=0x7fa2500fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:13.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.577+0000 7fa257819700 1 --2- 172.21.15.153:0/3907732015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2501038f0 0x7fa2500fcf60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa24c004940 tx=0x7fa24c009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:13.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.577+0000 7fa257819700 1 -- 172.21.15.153:0/3907732015 shutdown_connections 2024-03-11T22:59:13.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.577+0000 7fa257819700 1 --2- 172.21.15.153:0/3907732015 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2501038f0 0x7fa2500fcf60 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:13.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.577+0000 7fa257819700 1 --2- 172.21.15.153:0/3907732015 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa250102f50 0x7fa250103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:13.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.577+0000 7fa257819700 1 -- 172.21.15.153:0/3907732015 >> 172.21.15.153:0/3907732015 conn(0x7fa2500f8ac0 msgr2=0x7fa2500faed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:13.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.577+0000 7fa257819700 1 -- 172.21.15.153:0/3907732015 shutdown_connections 2024-03-11T22:59:13.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.577+0000 7fa257819700 1 -- 172.21.15.153:0/3907732015 wait complete. 2024-03-11T22:59:13.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.578+0000 7fa257819700 1 Processor -- start 2024-03-11T22:59:13.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.578+0000 7fa257819700 1 -- start start 2024-03-11T22:59:13.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.578+0000 7fa257819700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa250102f50 0x7fa250113b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:13.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.578+0000 7fa257819700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2501038f0 0x7fa250114070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:13.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.578+0000 7fa257819700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa250114750 con 0x7fa2501038f0 2024-03-11T22:59:13.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.578+0000 7fa257819700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25010d9f0 con 0x7fa250102f50 2024-03-11T22:59:13.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.578+0000 7fa254db4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2501038f0 0x7fa250114070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:13.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.578+0000 7fa254db4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2501038f0 0x7fa250114070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38272/0 (socket says 172.21.15.153:38272) 2024-03-11T22:59:13.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.578+0000 7fa2555b5700 1 --2- 172.21.15.153:0/383664180 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa250102f50 0x7fa250113b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:13.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.578+0000 7fa254db4700 1 -- 172.21.15.153:0/383664180 learned_addr learned my addr 172.21.15.153:0/383664180 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:59:13.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.578+0000 7fa254db4700 1 -- 172.21.15.153:0/383664180 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa250102f50 msgr2=0x7fa250113b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:13.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.579+0000 7fa254db4700 1 --2- 172.21.15.153:0/383664180 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa250102f50 0x7fa250113b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:13.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.579+0000 7fa254db4700 1 -- 172.21.15.153:0/383664180 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa24c005040 con 0x7fa2501038f0 2024-03-11T22:59:13.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.579+0000 7fa254db4700 1 --2- 172.21.15.153:0/383664180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2501038f0 0x7fa250114070 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa24c009100 tx=0x7fa24c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:13.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.579+0000 7fa245ffb700 1 -- 172.21.15.153:0/383664180 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa24c00e040 con 0x7fa2501038f0 2024-03-11T22:59:13.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.579+0000 7fa245ffb700 1 -- 172.21.15.153:0/383664180 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa24c004290 con 0x7fa2501038f0 2024-03-11T22:59:13.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.579+0000 7fa257819700 1 -- 172.21.15.153:0/383664180 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa25010dc70 con 0x7fa2501038f0 2024-03-11T22:59:13.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.579+0000 7fa245ffb700 1 -- 172.21.15.153:0/383664180 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa24c01c3f0 con 0x7fa2501038f0 2024-03-11T22:59:13.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.579+0000 7fa257819700 1 -- 172.21.15.153:0/383664180 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa25010e1b0 con 0x7fa2501038f0 2024-03-11T22:59:13.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.580+0000 7fa257819700 1 -- 172.21.15.153:0/383664180 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa25010e650 con 0x7fa2501038f0 2024-03-11T22:59:13.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.580+0000 7fa245ffb700 1 -- 172.21.15.153:0/383664180 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fa24c00bbf0 con 0x7fa2501038f0 2024-03-11T22:59:13.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.581+0000 7fa245ffb700 1 --2- 172.21.15.153:0/383664180 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fa23c0692b0 0x7fa23c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:13.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.581+0000 7fa245ffb700 1 -- 172.21.15.153:0/383664180 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa24c0880b0 con 0x7fa2501038f0 2024-03-11T22:59:13.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.581+0000 7fa2555b5700 1 --2- 172.21.15.153:0/383664180 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fa23c0692b0 0x7fa23c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:13.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.581+0000 7fa245ffb700 1 --2- 172.21.15.153:0/383664180 >> [v2:172.21.15.163:6800/3768791983,v1:172.21.15.163:6801/3768791983] conn(0x7fa23c06ef80 0x7fa23c071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:13.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.581+0000 7fa245ffb700 1 -- 172.21.15.153:0/383664180 --> [v2:172.21.15.163:6800/3768791983,v1:172.21.15.163:6801/3768791983] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa23c071a40 con 0x7fa23c06ef80 2024-03-11T22:59:13.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.581+0000 7fa245ffb700 1 -- 172.21.15.153:0/383664180 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa24c08a6a0 con 0x7fa2501038f0 2024-03-11T22:59:13.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.581+0000 7fa2555b5700 1 --2- 172.21.15.153:0/383664180 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fa23c0692b0 0x7fa23c06b760 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa240009bf0 tx=0x7fa2400091a0 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:13.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.581+0000 7fa255db6700 1 --2- 172.21.15.153:0/383664180 >> [v2:172.21.15.163:6800/3768791983,v1:172.21.15.163:6801/3768791983] conn(0x7fa23c06ef80 0x7fa23c071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:13.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.582+0000 7fa255db6700 1 --2- 172.21.15.153:0/383664180 >> [v2:172.21.15.163:6800/3768791983,v1:172.21.15.163:6801/3768791983] conn(0x7fa23c06ef80 0x7fa23c071390 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:13.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.583+0000 7fa245ffb700 1 -- 172.21.15.153:0/383664180 <== osd.3 v2:172.21.15.163:6800/3768791983 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fa23c071a40 con 0x7fa23c06ef80 2024-03-11T22:59:13.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.597+0000 7fa257819700 1 -- 172.21.15.153:0/383664180 --> [v2:172.21.15.163:6800/3768791983,v1:172.21.15.163:6801/3768791983] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa25005e690 con 0x7fa23c06ef80 2024-03-11T22:59:13.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.598+0000 7fa245ffb700 1 -- 172.21.15.153:0/383664180 <== osd.3 v2:172.21.15.163:6800/3768791983 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fa25005e690 con 0x7fa23c06ef80 2024-03-11T22:59:13.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.598+0000 7fa257819700 1 -- 172.21.15.153:0/383664180 >> [v2:172.21.15.163:6800/3768791983,v1:172.21.15.163:6801/3768791983] conn(0x7fa23c06ef80 msgr2=0x7fa23c071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:13.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.598+0000 7fa257819700 1 --2- 172.21.15.153:0/383664180 >> [v2:172.21.15.163:6800/3768791983,v1:172.21.15.163:6801/3768791983] conn(0x7fa23c06ef80 0x7fa23c071390 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:13.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.598+0000 7fa257819700 1 -- 172.21.15.153:0/383664180 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fa23c0692b0 msgr2=0x7fa23c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:13.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.598+0000 7fa257819700 1 --2- 172.21.15.153:0/383664180 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fa23c0692b0 0x7fa23c06b760 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa240009bf0 tx=0x7fa2400091a0 comp rx=0 tx=0).stop 2024-03-11T22:59:13.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.598+0000 7fa257819700 1 -- 172.21.15.153:0/383664180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2501038f0 msgr2=0x7fa250114070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:13.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.598+0000 7fa257819700 1 --2- 172.21.15.153:0/383664180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2501038f0 0x7fa250114070 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa24c009100 tx=0x7fa24c009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:13.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.598+0000 7fa257819700 1 -- 172.21.15.153:0/383664180 shutdown_connections 2024-03-11T22:59:13.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.598+0000 7fa257819700 1 --2- 172.21.15.153:0/383664180 >> [v2:172.21.15.163:6800/3768791983,v1:172.21.15.163:6801/3768791983] conn(0x7fa23c06ef80 0x7fa23c071390 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:13.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.598+0000 7fa257819700 1 --2- 172.21.15.153:0/383664180 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fa23c0692b0 0x7fa23c06b760 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:13.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.598+0000 7fa257819700 1 --2- 172.21.15.153:0/383664180 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa250102f50 0x7fa250113b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:13.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.598+0000 7fa257819700 1 --2- 172.21.15.153:0/383664180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2501038f0 0x7fa250114070 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:13.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.598+0000 7fa257819700 1 -- 172.21.15.153:0/383664180 >> 172.21.15.153:0/383664180 conn(0x7fa2500f8ac0 msgr2=0x7fa2500fa5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:13.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.599+0000 7fa257819700 1 -- 172.21.15.153:0/383664180 shutdown_connections 2024-03-11T22:59:13.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:13.599+0000 7fa257819700 1 -- 172.21.15.153:0/383664180 wait complete. 2024-03-11T22:59:13.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:13 smithi163 ceph-mon[35584]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:14.209 INFO:teuthology.orchestra.run.smithi153.stdout:115964117012 2024-03-11T22:59:14.210 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-11T22:59:15.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:15 smithi153 ceph-mon[28962]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:15.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:15 smithi163 ceph-mon[35584]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:15.802 INFO:teuthology.orchestra.run.smithi153.stdout:81604378649 2024-03-11T22:59:15.802 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-11T22:59:16.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.140+0000 7fb07af5f700 1 -- 172.21.15.153:0/197772706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0741027e0 msgr2=0x7fb0740fbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:16.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.140+0000 7fb07af5f700 1 --2- 172.21.15.153:0/197772706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0741027e0 0x7fb0740fbea0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb060004940 tx=0x7fb060009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:16.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.140+0000 7fb07af5f700 1 -- 172.21.15.153:0/197772706 shutdown_connections 2024-03-11T22:59:16.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.140+0000 7fb07af5f700 1 --2- 172.21.15.153:0/197772706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb0741027e0 0x7fb0740fbea0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:16.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.140+0000 7fb07af5f700 1 --2- 172.21.15.153:0/197772706 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb074101e40 0x7fb074102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:16.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.140+0000 7fb07af5f700 1 -- 172.21.15.153:0/197772706 >> 172.21.15.153:0/197772706 conn(0x7fb074071d90 msgr2=0x7fb074072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:16.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.140+0000 7fb07af5f700 1 -- 172.21.15.153:0/197772706 shutdown_connections 2024-03-11T22:59:16.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.140+0000 7fb07af5f700 1 -- 172.21.15.153:0/197772706 wait complete. 2024-03-11T22:59:16.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.140+0000 7fb07af5f700 1 Processor -- start 2024-03-11T22:59:16.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.140+0000 7fb07af5f700 1 -- start start 2024-03-11T22:59:16.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.141+0000 7fb07af5f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb074101e40 0x7fb07410eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:16.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.141+0000 7fb07af5f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0741027e0 0x7fb07410f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:16.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.141+0000 7fb07af5f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07410f680 con 0x7fb074101e40 2024-03-11T22:59:16.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.141+0000 7fb07af5f700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07410f7f0 con 0x7fb0741027e0 2024-03-11T22:59:16.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.141+0000 7fb078cfb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb074101e40 0x7fb07410eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:16.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.141+0000 7fb078cfb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb074101e40 0x7fb07410eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38296/0 (socket says 172.21.15.153:38296) 2024-03-11T22:59:16.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.141+0000 7fb078cfb700 1 -- 172.21.15.153:0/3604297701 learned_addr learned my addr 172.21.15.153:0/3604297701 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:59:16.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.141+0000 7fb078cfb700 1 -- 172.21.15.153:0/3604297701 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0741027e0 msgr2=0x7fb07410f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:16.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.141+0000 7fb078cfb700 1 --2- 172.21.15.153:0/3604297701 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0741027e0 0x7fb07410f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:16.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.141+0000 7fb078cfb700 1 -- 172.21.15.153:0/3604297701 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb060005040 con 0x7fb074101e40 2024-03-11T22:59:16.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.141+0000 7fb078cfb700 1 --2- 172.21.15.153:0/3604297701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb074101e40 0x7fb07410eaf0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb06800c860 tx=0x7fb068004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:16.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.141+0000 7fb0717fa700 1 -- 172.21.15.153:0/3604297701 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb06800a760 con 0x7fb074101e40 2024-03-11T22:59:16.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.141+0000 7fb0717fa700 1 -- 172.21.15.153:0/3604297701 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb06800ad90 con 0x7fb074101e40 2024-03-11T22:59:16.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.141+0000 7fb07af5f700 1 -- 172.21.15.153:0/3604297701 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb074117ac0 con 0x7fb074101e40 2024-03-11T22:59:16.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.141+0000 7fb0717fa700 1 -- 172.21.15.153:0/3604297701 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0680113a0 con 0x7fb074101e40 2024-03-11T22:59:16.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.141+0000 7fb07af5f700 1 -- 172.21.15.153:0/3604297701 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb074118000 con 0x7fb074101e40 2024-03-11T22:59:16.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.142+0000 7fb07af5f700 1 -- 172.21.15.153:0/3604297701 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb07405e610 con 0x7fb074101e40 2024-03-11T22:59:16.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.143+0000 7fb0717fa700 1 -- 172.21.15.153:0/3604297701 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fb068012400 con 0x7fb074101e40 2024-03-11T22:59:16.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.143+0000 7fb0717fa700 1 --2- 172.21.15.153:0/3604297701 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fb0640692a0 0x7fb06406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:16.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.143+0000 7fb073fff700 1 --2- 172.21.15.153:0/3604297701 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fb0640692a0 0x7fb06406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:16.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.143+0000 7fb0717fa700 1 -- 172.21.15.153:0/3604297701 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb0680878c0 con 0x7fb074101e40 2024-03-11T22:59:16.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.143+0000 7fb0717fa700 1 --2- 172.21.15.153:0/3604297701 >> [v2:172.21.15.163:6808/4243752277,v1:172.21.15.163:6809/4243752277] conn(0x7fb06406ef70 0x7fb064071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:16.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.143+0000 7fb0717fa700 1 -- 172.21.15.153:0/3604297701 --> [v2:172.21.15.163:6808/4243752277,v1:172.21.15.163:6809/4243752277] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb064071a30 con 0x7fb06406ef70 2024-03-11T22:59:16.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.143+0000 7fb0717fa700 1 -- 172.21.15.153:0/3604297701 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb068087ca0 con 0x7fb074101e40 2024-03-11T22:59:16.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.144+0000 7fb073fff700 1 --2- 172.21.15.153:0/3604297701 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fb0640692a0 0x7fb06406b750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb060004f90 tx=0x7fb060009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:16.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.144+0000 7fb0794fc700 1 --2- 172.21.15.153:0/3604297701 >> [v2:172.21.15.163:6808/4243752277,v1:172.21.15.163:6809/4243752277] conn(0x7fb06406ef70 0x7fb064071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:16.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.144+0000 7fb0794fc700 1 --2- 172.21.15.153:0/3604297701 >> [v2:172.21.15.163:6808/4243752277,v1:172.21.15.163:6809/4243752277] conn(0x7fb06406ef70 0x7fb064071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:16.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.145+0000 7fb0717fa700 1 -- 172.21.15.153:0/3604297701 <== osd.4 v2:172.21.15.163:6808/4243752277 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fb064071a30 con 0x7fb06406ef70 2024-03-11T22:59:16.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.160+0000 7fb07af5f700 1 -- 172.21.15.153:0/3604297701 --> [v2:172.21.15.163:6808/4243752277,v1:172.21.15.163:6809/4243752277] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb07405ea90 con 0x7fb06406ef70 2024-03-11T22:59:16.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.160+0000 7fb0717fa700 1 -- 172.21.15.153:0/3604297701 <== osd.4 v2:172.21.15.163:6808/4243752277 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb07405ea90 con 0x7fb06406ef70 2024-03-11T22:59:16.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.161+0000 7fb07af5f700 1 -- 172.21.15.153:0/3604297701 >> [v2:172.21.15.163:6808/4243752277,v1:172.21.15.163:6809/4243752277] conn(0x7fb06406ef70 msgr2=0x7fb064071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:16.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.161+0000 7fb07af5f700 1 --2- 172.21.15.153:0/3604297701 >> [v2:172.21.15.163:6808/4243752277,v1:172.21.15.163:6809/4243752277] conn(0x7fb06406ef70 0x7fb064071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:16.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.161+0000 7fb07af5f700 1 -- 172.21.15.153:0/3604297701 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fb0640692a0 msgr2=0x7fb06406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:16.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.161+0000 7fb07af5f700 1 --2- 172.21.15.153:0/3604297701 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fb0640692a0 0x7fb06406b750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb060004f90 tx=0x7fb060009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:16.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.161+0000 7fb07af5f700 1 -- 172.21.15.153:0/3604297701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb074101e40 msgr2=0x7fb07410eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:16.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.161+0000 7fb07af5f700 1 --2- 172.21.15.153:0/3604297701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb074101e40 0x7fb07410eaf0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb06800c860 tx=0x7fb068004470 comp rx=0 tx=0).stop 2024-03-11T22:59:16.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.161+0000 7fb07af5f700 1 -- 172.21.15.153:0/3604297701 shutdown_connections 2024-03-11T22:59:16.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.161+0000 7fb07af5f700 1 --2- 172.21.15.153:0/3604297701 >> [v2:172.21.15.163:6808/4243752277,v1:172.21.15.163:6809/4243752277] conn(0x7fb06406ef70 0x7fb064071380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:16.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.161+0000 7fb07af5f700 1 --2- 172.21.15.153:0/3604297701 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb074101e40 0x7fb07410eaf0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:16.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.161+0000 7fb07af5f700 1 --2- 172.21.15.153:0/3604297701 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fb0640692a0 0x7fb06406b750 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:16.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.161+0000 7fb07af5f700 1 --2- 172.21.15.153:0/3604297701 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0741027e0 0x7fb07410f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:16.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.161+0000 7fb07af5f700 1 -- 172.21.15.153:0/3604297701 >> 172.21.15.153:0/3604297701 conn(0x7fb074071d90 msgr2=0x7fb0740fae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:16.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.161+0000 7fb07af5f700 1 -- 172.21.15.153:0/3604297701 shutdown_connections 2024-03-11T22:59:16.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:15.161+0000 7fb07af5f700 1 -- 172.21.15.153:0/3604297701 wait complete. 2024-03-11T22:59:16.365 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:59:16.767 INFO:teuthology.orchestra.run.smithi153.stdout:150323855375 2024-03-11T22:59:16.767 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-11T22:59:16.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.935+0000 7f1a4c0d8700 1 -- 172.21.15.153:0/3397153884 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a44102880 msgr2=0x7f1a440fc680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:16.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.935+0000 7f1a4c0d8700 1 --2- 172.21.15.153:0/3397153884 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a44102880 0x7f1a440fc680 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1a40004940 tx=0x7f1a40009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:16.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.935+0000 7f1a4c0d8700 1 -- 172.21.15.153:0/3397153884 shutdown_connections 2024-03-11T22:59:16.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.935+0000 7f1a4c0d8700 1 --2- 172.21.15.153:0/3397153884 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a44102880 0x7f1a440fc680 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:16.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.935+0000 7f1a4c0d8700 1 --2- 172.21.15.153:0/3397153884 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a44101ee0 0x7f1a441022b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:16.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.935+0000 7f1a4c0d8700 1 -- 172.21.15.153:0/3397153884 >> 172.21.15.153:0/3397153884 conn(0x7f1a44071560 msgr2=0x7f1a44071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:16.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.935+0000 7f1a4c0d8700 1 -- 172.21.15.153:0/3397153884 shutdown_connections 2024-03-11T22:59:16.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.935+0000 7f1a4c0d8700 1 -- 172.21.15.153:0/3397153884 wait complete. 2024-03-11T22:59:16.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.936+0000 7f1a4c0d8700 1 Processor -- start 2024-03-11T22:59:16.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.936+0000 7f1a4c0d8700 1 -- start start 2024-03-11T22:59:16.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.936+0000 7f1a4c0d8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a44101ee0 0x7f1a44115c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:16.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.936+0000 7f1a4c0d8700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a44102880 0x7f1a441161b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:16.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.936+0000 7f1a4c0d8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a44116890 con 0x7f1a44101ee0 2024-03-11T22:59:16.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.936+0000 7f1a4c0d8700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4410fb30 con 0x7f1a44102880 2024-03-11T22:59:16.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.936+0000 7f1a49e74700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a44101ee0 0x7f1a44115c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:16.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.936+0000 7f1a49e74700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a44101ee0 0x7f1a44115c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38310/0 (socket says 172.21.15.153:38310) 2024-03-11T22:59:16.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.936+0000 7f1a49e74700 1 -- 172.21.15.153:0/1667931824 learned_addr learned my addr 172.21.15.153:0/1667931824 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:59:16.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.936+0000 7f1a49673700 1 --2- 172.21.15.153:0/1667931824 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a44102880 0x7f1a441161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:16.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.936+0000 7f1a49e74700 1 -- 172.21.15.153:0/1667931824 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a44102880 msgr2=0x7f1a441161b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:16.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.936+0000 7f1a49e74700 1 --2- 172.21.15.153:0/1667931824 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a44102880 0x7f1a441161b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:16.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.936+0000 7f1a49e74700 1 -- 172.21.15.153:0/1667931824 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a40005040 con 0x7f1a44101ee0 2024-03-11T22:59:16.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.936+0000 7f1a49673700 1 --2- 172.21.15.153:0/1667931824 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a44102880 0x7f1a441161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T22:59:16.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.937+0000 7f1a49e74700 1 --2- 172.21.15.153:0/1667931824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a44101ee0 0x7f1a44115c70 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1a3400b790 tx=0x7f1a34009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:16.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.937+0000 7f1a3a7fc700 1 -- 172.21.15.153:0/1667931824 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a3400f770 con 0x7f1a44101ee0 2024-03-11T22:59:16.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.937+0000 7f1a3a7fc700 1 -- 172.21.15.153:0/1667931824 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1a3400fda0 con 0x7f1a44101ee0 2024-03-11T22:59:16.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.937+0000 7f1a4c0d8700 1 -- 172.21.15.153:0/1667931824 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a4410fe10 con 0x7f1a44101ee0 2024-03-11T22:59:16.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.937+0000 7f1a3a7fc700 1 -- 172.21.15.153:0/1667931824 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a3400d310 con 0x7f1a44101ee0 2024-03-11T22:59:16.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.937+0000 7f1a4c0d8700 1 -- 172.21.15.153:0/1667931824 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a44110350 con 0x7f1a44101ee0 2024-03-11T22:59:16.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.937+0000 7f1a4c0d8700 1 -- 172.21.15.153:0/1667931824 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1a441108d0 con 0x7f1a44101ee0 2024-03-11T22:59:16.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.938+0000 7f1a3a7fc700 1 -- 172.21.15.153:0/1667931824 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f1a34010420 con 0x7f1a44101ee0 2024-03-11T22:59:16.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.939+0000 7f1a3a7fc700 1 --2- 172.21.15.153:0/1667931824 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f1a300692a0 0x7f1a3006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:16.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.939+0000 7f1a49673700 1 --2- 172.21.15.153:0/1667931824 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f1a300692a0 0x7f1a3006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:16.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.939+0000 7f1a3a7fc700 1 -- 172.21.15.153:0/1667931824 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1a34087930 con 0x7f1a44101ee0 2024-03-11T22:59:16.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.939+0000 7f1a3a7fc700 1 --2- 172.21.15.153:0/1667931824 >> [v2:172.21.15.153:6810/2718652372,v1:172.21.15.153:6811/2718652372] conn(0x7f1a3006ef70 0x7f1a30071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:16.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.939+0000 7f1a3a7fc700 1 -- 172.21.15.153:0/1667931824 --> [v2:172.21.15.153:6810/2718652372,v1:172.21.15.153:6811/2718652372] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1a30071a30 con 0x7f1a3006ef70 2024-03-11T22:59:16.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.939+0000 7f1a3a7fc700 1 -- 172.21.15.153:0/1667931824 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1a34087d10 con 0x7f1a44101ee0 2024-03-11T22:59:16.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.939+0000 7f1a4a675700 1 --2- 172.21.15.153:0/1667931824 >> [v2:172.21.15.153:6810/2718652372,v1:172.21.15.153:6811/2718652372] conn(0x7f1a3006ef70 0x7f1a30071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:16.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.939+0000 7f1a49673700 1 --2- 172.21.15.153:0/1667931824 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f1a300692a0 0x7f1a3006b750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1a441114a0 tx=0x7f1a40009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:16.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.940+0000 7f1a4a675700 1 --2- 172.21.15.153:0/1667931824 >> [v2:172.21.15.153:6810/2718652372,v1:172.21.15.153:6811/2718652372] conn(0x7f1a3006ef70 0x7f1a30071380 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:16.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.941+0000 7f1a3a7fc700 1 -- 172.21.15.153:0/1667931824 <== osd.1 v2:172.21.15.153:6810/2718652372 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f1a30071a30 con 0x7f1a3006ef70 2024-03-11T22:59:16.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.956+0000 7f1a4c0d8700 1 -- 172.21.15.153:0/1667931824 --> [v2:172.21.15.153:6810/2718652372,v1:172.21.15.153:6811/2718652372] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1a440face0 con 0x7f1a3006ef70 2024-03-11T22:59:16.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.956+0000 7f1a3a7fc700 1 -- 172.21.15.153:0/1667931824 <== osd.1 v2:172.21.15.153:6810/2718652372 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1a440face0 con 0x7f1a3006ef70 2024-03-11T22:59:16.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.957+0000 7f1a4c0d8700 1 -- 172.21.15.153:0/1667931824 >> [v2:172.21.15.153:6810/2718652372,v1:172.21.15.153:6811/2718652372] conn(0x7f1a3006ef70 msgr2=0x7f1a30071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:16.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.957+0000 7f1a4c0d8700 1 --2- 172.21.15.153:0/1667931824 >> [v2:172.21.15.153:6810/2718652372,v1:172.21.15.153:6811/2718652372] conn(0x7f1a3006ef70 0x7f1a30071380 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:16.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.957+0000 7f1a4c0d8700 1 -- 172.21.15.153:0/1667931824 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f1a300692a0 msgr2=0x7f1a3006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:16.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.957+0000 7f1a4c0d8700 1 --2- 172.21.15.153:0/1667931824 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f1a300692a0 0x7f1a3006b750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1a441114a0 tx=0x7f1a40009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:16.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.957+0000 7f1a4c0d8700 1 -- 172.21.15.153:0/1667931824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a44101ee0 msgr2=0x7f1a44115c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:16.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.957+0000 7f1a4c0d8700 1 --2- 172.21.15.153:0/1667931824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a44101ee0 0x7f1a44115c70 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1a3400b790 tx=0x7f1a34009250 comp rx=0 tx=0).stop 2024-03-11T22:59:16.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.957+0000 7f1a4c0d8700 1 -- 172.21.15.153:0/1667931824 shutdown_connections 2024-03-11T22:59:16.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.957+0000 7f1a4c0d8700 1 --2- 172.21.15.153:0/1667931824 >> [v2:172.21.15.153:6810/2718652372,v1:172.21.15.153:6811/2718652372] conn(0x7f1a3006ef70 0x7f1a30071380 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:16.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.957+0000 7f1a4c0d8700 1 --2- 172.21.15.153:0/1667931824 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1a44101ee0 0x7f1a44115c70 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:16.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.957+0000 7f1a4c0d8700 1 --2- 172.21.15.153:0/1667931824 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f1a300692a0 0x7f1a3006b750 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:16.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.957+0000 7f1a4c0d8700 1 --2- 172.21.15.153:0/1667931824 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1a44102880 0x7f1a441161b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:16.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.957+0000 7f1a4c0d8700 1 -- 172.21.15.153:0/1667931824 >> 172.21.15.153:0/1667931824 conn(0x7f1a44071560 msgr2=0x7f1a440fa310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:16.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.957+0000 7f1a4c0d8700 1 -- 172.21.15.153:0/1667931824 shutdown_connections 2024-03-11T22:59:16.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:16.957+0000 7f1a4c0d8700 1 -- 172.21.15.153:0/1667931824 wait complete. 2024-03-11T22:59:17.211 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:59:17.607 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:17 smithi153 ceph-mon[28962]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:17.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:17 smithi163 ceph-mon[35584]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:18.031 INFO:teuthology.orchestra.run.smithi153.stdout:60129542175 2024-03-11T22:59:18.031 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-11T22:59:18.040 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:59:18.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:18 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:59:18.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:18 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:59:19.674 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:19 smithi153 ceph-mon[28962]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:19.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:19 smithi163 ceph-mon[35584]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:19.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:19.967+0000 7f5f0f375700 1 -- 172.21.15.153:0/515605762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f080fe8f0 msgr2=0x7f5f08107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:19.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:19.967+0000 7f5f0f375700 1 --2- 172.21.15.153:0/515605762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f080fe8f0 0x7f5f08107050 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8004970 tx=0x7f5ef8009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:19.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:19.967+0000 7f5f0f375700 1 -- 172.21.15.153:0/515605762 shutdown_connections 2024-03-11T22:59:19.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:19.967+0000 7f5f0f375700 1 --2- 172.21.15.153:0/515605762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f080fe8f0 0x7f5f08107050 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:19.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:19.967+0000 7f5f0f375700 1 --2- 172.21.15.153:0/515605762 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5f080fdfe0 0x7f5f080fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:19.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:19.967+0000 7f5f0f375700 1 -- 172.21.15.153:0/515605762 >> 172.21.15.153:0/515605762 conn(0x7f5f08071d90 msgr2=0x7f5f08072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:19.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:19.967+0000 7f5f0f375700 1 -- 172.21.15.153:0/515605762 shutdown_connections 2024-03-11T22:59:19.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:19.967+0000 7f5f0f375700 1 -- 172.21.15.153:0/515605762 wait complete. 2024-03-11T22:59:19.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:19.968+0000 7f5f0f375700 1 Processor -- start 2024-03-11T22:59:19.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:19.968+0000 7f5f0f375700 1 -- start start 2024-03-11T22:59:19.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:19.968+0000 7f5f0f375700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f080fdfe0 0x7f5f08113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:19.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:19.968+0000 7f5f0f375700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5f080fe8f0 0x7f5f08113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:19.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:19.968+0000 7f5f0f375700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f08114690 con 0x7f5f080fdfe0 2024-03-11T22:59:19.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:19.968+0000 7f5f0f375700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0810d930 con 0x7f5f080fe8f0 2024-03-11T22:59:19.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:19.968+0000 7f5f0d111700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f080fdfe0 0x7f5f08113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:19.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:19.968+0000 7f5f0d111700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f080fdfe0 0x7f5f08113a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38332/0 (socket says 172.21.15.153:38332) 2024-03-11T22:59:19.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:19.968+0000 7f5f0d111700 1 -- 172.21.15.153:0/3759600123 learned_addr learned my addr 172.21.15.153:0/3759600123 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:59:19.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:19.968+0000 7f5f0c910700 1 --2- 172.21.15.153:0/3759600123 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5f080fe8f0 0x7f5f08113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:19.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:19.968+0000 7f5f0d111700 1 -- 172.21.15.153:0/3759600123 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5f080fe8f0 msgr2=0x7f5f08113fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:19.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:19.969+0000 7f5f0d111700 1 --2- 172.21.15.153:0/3759600123 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5f080fe8f0 0x7f5f08113fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:19.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:19.969+0000 7f5f0d111700 1 -- 172.21.15.153:0/3759600123 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef8005040 con 0x7f5f080fdfe0 2024-03-11T22:59:19.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:19.969+0000 7f5f0d111700 1 --2- 172.21.15.153:0/3759600123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f080fdfe0 0x7f5f08113a70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5f0400b790 tx=0x7f5f04009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:19.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:19.969+0000 7f5efdffb700 1 -- 172.21.15.153:0/3759600123 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f0400f770 con 0x7f5f080fdfe0 2024-03-11T22:59:19.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:19.969+0000 7f5efdffb700 1 -- 172.21.15.153:0/3759600123 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5f0400fda0 con 0x7f5f080fdfe0 2024-03-11T22:59:19.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:19.969+0000 7f5efdffb700 1 -- 172.21.15.153:0/3759600123 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5f0400d310 con 0x7f5f080fdfe0 2024-03-11T22:59:19.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:19.969+0000 7f5f0f375700 1 -- 172.21.15.153:0/3759600123 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f0810dc10 con 0x7f5f080fdfe0 2024-03-11T22:59:19.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:19.969+0000 7f5f0f375700 1 -- 172.21.15.153:0/3759600123 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f0810e120 con 0x7f5f080fdfe0 2024-03-11T22:59:19.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:19.970+0000 7f5f0f375700 1 -- 172.21.15.153:0/3759600123 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f081046c0 con 0x7f5f080fdfe0 2024-03-11T22:59:19.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:19.970+0000 7f5efdffb700 1 -- 172.21.15.153:0/3759600123 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f5f04010420 con 0x7f5f080fdfe0 2024-03-11T22:59:19.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:19.971+0000 7f5efdffb700 1 --2- 172.21.15.153:0/3759600123 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f5ef40692a0 0x7f5ef406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:19.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:19.971+0000 7f5f0c910700 1 --2- 172.21.15.153:0/3759600123 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f5ef40692a0 0x7f5ef406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:19.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:19.971+0000 7f5efdffb700 1 -- 172.21.15.153:0/3759600123 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5f04086e70 con 0x7f5f080fdfe0 2024-03-11T22:59:19.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:19.971+0000 7f5f0c910700 1 --2- 172.21.15.153:0/3759600123 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f5ef40692a0 0x7f5ef406b750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5f0810ec60 tx=0x7f5ef8009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:19.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:19.976+0000 7f5efdffb700 1 -- 172.21.15.153:0/3759600123 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f5f04056f70 con 0x7f5f080fdfe0 2024-03-11T22:59:20.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:20.166+0000 7f5f0f375700 1 -- 172.21.15.153:0/3759600123 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f5f080738b0 con 0x7f5f080fdfe0 2024-03-11T22:59:20.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:20.167+0000 7f5efdffb700 1 -- 172.21.15.153:0/3759600123 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5f04056b00 con 0x7f5f080fdfe0 2024-03-11T22:59:20.171 INFO:teuthology.orchestra.run.smithi153.stdout:38654705700 2024-03-11T22:59:20.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:20.169+0000 7f5f0f375700 1 -- 172.21.15.153:0/3759600123 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f5ef40692a0 msgr2=0x7f5ef406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:20.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:20.169+0000 7f5f0f375700 1 --2- 172.21.15.153:0/3759600123 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f5ef40692a0 0x7f5ef406b750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5f0810ec60 tx=0x7f5ef8009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:20.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:20.169+0000 7f5f0f375700 1 -- 172.21.15.153:0/3759600123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f080fdfe0 msgr2=0x7f5f08113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:20.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:20.169+0000 7f5f0f375700 1 --2- 172.21.15.153:0/3759600123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f080fdfe0 0x7f5f08113a70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5f0400b790 tx=0x7f5f04009250 comp rx=0 tx=0).stop 2024-03-11T22:59:20.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:20.169+0000 7f5f0f375700 1 -- 172.21.15.153:0/3759600123 shutdown_connections 2024-03-11T22:59:20.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:20.169+0000 7f5f0f375700 1 --2- 172.21.15.153:0/3759600123 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5f080fdfe0 0x7f5f08113a70 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:20.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:20.169+0000 7f5f0f375700 1 --2- 172.21.15.153:0/3759600123 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f5ef40692a0 0x7f5ef406b750 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:20.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:20.169+0000 7f5f0f375700 1 --2- 172.21.15.153:0/3759600123 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5f080fe8f0 0x7f5f08113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:20.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:20.169+0000 7f5f0f375700 1 -- 172.21.15.153:0/3759600123 >> 172.21.15.153:0/3759600123 conn(0x7f5f08071d90 msgr2=0x7f5f08101840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:20.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:20.169+0000 7f5f0f375700 1 -- 172.21.15.153:0/3759600123 shutdown_connections 2024-03-11T22:59:20.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:20.169+0000 7f5f0f375700 1 -- 172.21.15.153:0/3759600123 wait complete. 2024-03-11T22:59:20.711 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:20 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/3759600123' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T22:59:20.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:20 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/3759600123' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T22:59:21.669 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:21 smithi153 ceph-mon[28962]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:21.695 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705698 got 38654705700 for osd.0 2024-03-11T22:59:21.696 DEBUG:teuthology.parallel:result is None 2024-03-11T22:59:21.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:21 smithi163 ceph-mon[35584]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:22.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:22.827+0000 7f419a2b8700 1 -- 172.21.15.153:0/344247419 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4194101e50 msgr2=0x7f4194102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:22.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:22.827+0000 7f419a2b8700 1 --2- 172.21.15.153:0/344247419 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4194101e50 0x7f4194102220 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f41840049c0 tx=0x7f4184009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:22.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:22.827+0000 7f419a2b8700 1 -- 172.21.15.153:0/344247419 shutdown_connections 2024-03-11T22:59:22.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:22.827+0000 7f419a2b8700 1 --2- 172.21.15.153:0/344247419 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41941027f0 0x7f41940fbe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:22.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:22.827+0000 7f419a2b8700 1 --2- 172.21.15.153:0/344247419 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4194101e50 0x7f4194102220 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:22.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:22.827+0000 7f419a2b8700 1 -- 172.21.15.153:0/344247419 >> 172.21.15.153:0/344247419 conn(0x7f41940f79e0 msgr2=0x7f41940f9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:22.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:22.827+0000 7f419a2b8700 1 -- 172.21.15.153:0/344247419 shutdown_connections 2024-03-11T22:59:22.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:22.827+0000 7f419a2b8700 1 -- 172.21.15.153:0/344247419 wait complete. 2024-03-11T22:59:22.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:22.828+0000 7f419a2b8700 1 Processor -- start 2024-03-11T22:59:22.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:22.828+0000 7f419a2b8700 1 -- start start 2024-03-11T22:59:22.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:22.828+0000 7f419a2b8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4194101e50 0x7f4194114980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:22.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:22.828+0000 7f419a2b8700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41941027f0 0x7f419410b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:22.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:22.828+0000 7f419a2b8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4194115010 con 0x7f4194101e50 2024-03-11T22:59:22.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:22.828+0000 7f419a2b8700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f419410bca0 con 0x7f41941027f0 2024-03-11T22:59:22.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:22.828+0000 7f4193fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4194101e50 0x7f4194114980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:22.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:22.828+0000 7f4193fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4194101e50 0x7f4194114980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:45120/0 (socket says 172.21.15.153:45120) 2024-03-11T22:59:22.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:22.828+0000 7f4193fff700 1 -- 172.21.15.153:0/2941685578 learned_addr learned my addr 172.21.15.153:0/2941685578 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:59:22.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:22.829+0000 7f41937fe700 1 --2- 172.21.15.153:0/2941685578 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41941027f0 0x7f419410b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:22.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:22.829+0000 7f4193fff700 1 -- 172.21.15.153:0/2941685578 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41941027f0 msgr2=0x7f419410b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:22.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:22.829+0000 7f4193fff700 1 --2- 172.21.15.153:0/2941685578 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41941027f0 0x7f419410b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:22.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:22.829+0000 7f4193fff700 1 -- 172.21.15.153:0/2941685578 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4184005040 con 0x7f4194101e50 2024-03-11T22:59:22.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:22.829+0000 7f4193fff700 1 --2- 172.21.15.153:0/2941685578 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4194101e50 0x7f4194114980 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4184009b10 tx=0x7f4184009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:22.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:22.830+0000 7f4190ff9700 1 -- 172.21.15.153:0/2941685578 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f418400e070 con 0x7f4194101e50 2024-03-11T22:59:22.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:22.830+0000 7f4190ff9700 1 -- 172.21.15.153:0/2941685578 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f418400bcc0 con 0x7f4194101e50 2024-03-11T22:59:22.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:22.830+0000 7f4190ff9700 1 -- 172.21.15.153:0/2941685578 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4184022640 con 0x7f4194101e50 2024-03-11T22:59:22.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:22.830+0000 7f419a2b8700 1 -- 172.21.15.153:0/2941685578 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f419410bf20 con 0x7f4194101e50 2024-03-11T22:59:22.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:22.830+0000 7f419a2b8700 1 -- 172.21.15.153:0/2941685578 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f419410c3a0 con 0x7f4194101e50 2024-03-11T22:59:22.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:22.831+0000 7f419a2b8700 1 -- 172.21.15.153:0/2941685578 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4194064b00 con 0x7f4194101e50 2024-03-11T22:59:22.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:22.832+0000 7f4190ff9700 1 -- 172.21.15.153:0/2941685578 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f418402b3f0 con 0x7f4194101e50 2024-03-11T22:59:22.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:22.832+0000 7f4190ff9700 1 --2- 172.21.15.153:0/2941685578 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f417c069360 0x7f417c06b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:22.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:22.832+0000 7f4190ff9700 1 -- 172.21.15.153:0/2941685578 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4184088370 con 0x7f4194101e50 2024-03-11T22:59:22.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:22.832+0000 7f41937fe700 1 --2- 172.21.15.153:0/2941685578 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f417c069360 0x7f417c06b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:22.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:22.833+0000 7f41937fe700 1 --2- 172.21.15.153:0/2941685578 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f417c069360 0x7f417c06b810 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f419410d160 tx=0x7f418800b040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:22.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:22.839+0000 7f4190ff9700 1 -- 172.21.15.153:0/2941685578 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f4184058580 con 0x7f4194101e50 2024-03-11T22:59:23.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:23.030+0000 7f419a2b8700 1 -- 172.21.15.153:0/2941685578 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f419404bed0 con 0x7f4194101e50 2024-03-11T22:59:23.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:23.030+0000 7f4190ff9700 1 -- 172.21.15.153:0/2941685578 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f4184058110 con 0x7f4194101e50 2024-03-11T22:59:23.035 INFO:teuthology.orchestra.run.smithi153.stdout:150323855377 2024-03-11T22:59:23.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:23.032+0000 7f419a2b8700 1 -- 172.21.15.153:0/2941685578 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f417c069360 msgr2=0x7f417c06b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:23.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:23.032+0000 7f419a2b8700 1 --2- 172.21.15.153:0/2941685578 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f417c069360 0x7f417c06b810 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f419410d160 tx=0x7f418800b040 comp rx=0 tx=0).stop 2024-03-11T22:59:23.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:23.032+0000 7f419a2b8700 1 -- 172.21.15.153:0/2941685578 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4194101e50 msgr2=0x7f4194114980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:23.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:23.032+0000 7f419a2b8700 1 --2- 172.21.15.153:0/2941685578 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4194101e50 0x7f4194114980 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4184009b10 tx=0x7f4184009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:23.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:23.032+0000 7f419a2b8700 1 -- 172.21.15.153:0/2941685578 shutdown_connections 2024-03-11T22:59:23.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:23.032+0000 7f419a2b8700 1 --2- 172.21.15.153:0/2941685578 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4194101e50 0x7f4194114980 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:23.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:23.032+0000 7f419a2b8700 1 --2- 172.21.15.153:0/2941685578 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f417c069360 0x7f417c06b810 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:23.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:23.032+0000 7f419a2b8700 1 --2- 172.21.15.153:0/2941685578 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41941027f0 0x7f419410b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:23.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:23.032+0000 7f419a2b8700 1 -- 172.21.15.153:0/2941685578 >> 172.21.15.153:0/2941685578 conn(0x7f41940f79e0 msgr2=0x7f41940f9460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:23.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:23.032+0000 7f419a2b8700 1 -- 172.21.15.153:0/2941685578 shutdown_connections 2024-03-11T22:59:23.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:23.032+0000 7f419a2b8700 1 -- 172.21.15.153:0/2941685578 wait complete. 2024-03-11T22:59:23.708 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:23 smithi153 ceph-mon[28962]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:23.709 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:23 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/2941685578' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T22:59:23.712 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:59:23.738 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855375 got 150323855377 for osd.4 2024-03-11T22:59:23.739 DEBUG:teuthology.parallel:result is None 2024-03-11T22:59:23.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:23 smithi163 ceph-mon[35584]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:23.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:23 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/2941685578' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T22:59:24.564 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:59:25.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.131+0000 7f7e528f9700 1 -- 172.21.15.153:0/255844151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e4c102800 msgr2=0x7f7e4c0fbe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:25.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.131+0000 7f7e528f9700 1 --2- 172.21.15.153:0/255844151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e4c102800 0x7f7e4c0fbe70 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7e40004940 tx=0x7f7e40009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:25.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.131+0000 7f7e528f9700 1 -- 172.21.15.153:0/255844151 shutdown_connections 2024-03-11T22:59:25.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.131+0000 7f7e528f9700 1 --2- 172.21.15.153:0/255844151 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e4c102800 0x7f7e4c0fbe70 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:25.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.131+0000 7f7e528f9700 1 --2- 172.21.15.153:0/255844151 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e4c101e60 0x7f7e4c102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:25.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.131+0000 7f7e528f9700 1 -- 172.21.15.153:0/255844151 >> 172.21.15.153:0/255844151 conn(0x7f7e4c0f7a10 msgr2=0x7f7e4c0f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:25.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.131+0000 7f7e528f9700 1 -- 172.21.15.153:0/255844151 shutdown_connections 2024-03-11T22:59:25.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.131+0000 7f7e528f9700 1 -- 172.21.15.153:0/255844151 wait complete. 2024-03-11T22:59:25.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.132+0000 7f7e528f9700 1 Processor -- start 2024-03-11T22:59:25.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.132+0000 7f7e528f9700 1 -- start start 2024-03-11T22:59:25.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.132+0000 7f7e528f9700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e4c101e60 0x7f7e4c0fdae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:25.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.132+0000 7f7e528f9700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e4c102800 0x7f7e4c0fe020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:25.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.132+0000 7f7e528f9700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4c0fe620 con 0x7f7e4c101e60 2024-03-11T22:59:25.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.132+0000 7f7e528f9700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4c0fe790 con 0x7f7e4c102800 2024-03-11T22:59:25.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.132+0000 7f7e4bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e4c101e60 0x7f7e4c0fdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:25.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.132+0000 7f7e4bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e4c101e60 0x7f7e4c0fdae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:45136/0 (socket says 172.21.15.153:45136) 2024-03-11T22:59:25.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.132+0000 7f7e4bfff700 1 -- 172.21.15.153:0/794661563 learned_addr learned my addr 172.21.15.153:0/794661563 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:59:25.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.132+0000 7f7e4b7fe700 1 --2- 172.21.15.153:0/794661563 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e4c102800 0x7f7e4c0fe020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:25.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.133+0000 7f7e4bfff700 1 -- 172.21.15.153:0/794661563 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e4c102800 msgr2=0x7f7e4c0fe020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:25.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.133+0000 7f7e4bfff700 1 --2- 172.21.15.153:0/794661563 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e4c102800 0x7f7e4c0fe020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:25.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.133+0000 7f7e4bfff700 1 -- 172.21.15.153:0/794661563 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e40005040 con 0x7f7e4c101e60 2024-03-11T22:59:25.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.133+0000 7f7e4bfff700 1 --2- 172.21.15.153:0/794661563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e4c101e60 0x7f7e4c0fdae0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c002a50 tx=0x7f7e3c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:25.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.133+0000 7f7e48ff9700 1 -- 172.21.15.153:0/794661563 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e3c007cc0 con 0x7f7e4c101e60 2024-03-11T22:59:25.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.133+0000 7f7e528f9700 1 -- 172.21.15.153:0/794661563 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e4c119d20 con 0x7f7e4c101e60 2024-03-11T22:59:25.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.133+0000 7f7e48ff9700 1 -- 172.21.15.153:0/794661563 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7e3c00f040 con 0x7f7e4c101e60 2024-03-11T22:59:25.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.133+0000 7f7e48ff9700 1 -- 172.21.15.153:0/794661563 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7e3c00da30 con 0x7f7e4c101e60 2024-03-11T22:59:25.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.133+0000 7f7e528f9700 1 -- 172.21.15.153:0/794661563 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e4c11a1d0 con 0x7f7e4c101e60 2024-03-11T22:59:25.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.134+0000 7f7e528f9700 1 -- 172.21.15.153:0/794661563 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e4c0fb610 con 0x7f7e4c101e60 2024-03-11T22:59:25.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.134+0000 7f7e48ff9700 1 -- 172.21.15.153:0/794661563 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f7e3c00db90 con 0x7f7e4c101e60 2024-03-11T22:59:25.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.135+0000 7f7e48ff9700 1 --2- 172.21.15.153:0/794661563 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f7e340692a0 0x7f7e3406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:25.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.135+0000 7f7e48ff9700 1 -- 172.21.15.153:0/794661563 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7e3c086c00 con 0x7f7e4c101e60 2024-03-11T22:59:25.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.135+0000 7f7e4b7fe700 1 --2- 172.21.15.153:0/794661563 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f7e340692a0 0x7f7e3406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:25.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.135+0000 7f7e4b7fe700 1 --2- 172.21.15.153:0/794661563 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f7e340692a0 0x7f7e3406b750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7e4000b3b0 tx=0x7f7e40009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:25.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.140+0000 7f7e48ff9700 1 -- 172.21.15.153:0/794661563 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f7e3c08c020 con 0x7f7e4c101e60 2024-03-11T22:59:25.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.329+0000 7f7e528f9700 1 -- 172.21.15.153:0/794661563 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f7e4c0ff320 con 0x7f7e4c101e60 2024-03-11T22:59:25.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.329+0000 7f7e48ff9700 1 -- 172.21.15.153:0/794661563 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f7e3c056c80 con 0x7f7e4c101e60 2024-03-11T22:59:25.334 INFO:teuthology.orchestra.run.smithi153.stdout:171798691851 2024-03-11T22:59:25.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.331+0000 7f7e528f9700 1 -- 172.21.15.153:0/794661563 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f7e340692a0 msgr2=0x7f7e3406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:25.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.331+0000 7f7e528f9700 1 --2- 172.21.15.153:0/794661563 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f7e340692a0 0x7f7e3406b750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f7e4000b3b0 tx=0x7f7e40009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:25.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.331+0000 7f7e528f9700 1 -- 172.21.15.153:0/794661563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e4c101e60 msgr2=0x7f7e4c0fdae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:25.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.331+0000 7f7e528f9700 1 --2- 172.21.15.153:0/794661563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e4c101e60 0x7f7e4c0fdae0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c002a50 tx=0x7f7e3c0046b0 comp rx=0 tx=0).stop 2024-03-11T22:59:25.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.331+0000 7f7e528f9700 1 -- 172.21.15.153:0/794661563 shutdown_connections 2024-03-11T22:59:25.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.331+0000 7f7e528f9700 1 --2- 172.21.15.153:0/794661563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e4c101e60 0x7f7e4c0fdae0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:25.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.331+0000 7f7e528f9700 1 --2- 172.21.15.153:0/794661563 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f7e340692a0 0x7f7e3406b750 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:25.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.331+0000 7f7e528f9700 1 --2- 172.21.15.153:0/794661563 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e4c102800 0x7f7e4c0fe020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:25.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.331+0000 7f7e528f9700 1 -- 172.21.15.153:0/794661563 >> 172.21.15.153:0/794661563 conn(0x7f7e4c0f7a10 msgr2=0x7f7e4c065ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:25.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.332+0000 7f7e528f9700 1 -- 172.21.15.153:0/794661563 shutdown_connections 2024-03-11T22:59:25.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.332+0000 7f7e528f9700 1 -- 172.21.15.153:0/794661563 wait complete. 2024-03-11T22:59:25.721 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:25 smithi153 ceph-mon[28962]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:25.722 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:25 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/794661563' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T22:59:25.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:25 smithi163 ceph-mon[35584]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:25.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:25 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/794661563' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T22:59:26.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:26 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/2881909208' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T22:59:26.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:26 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/2881909208' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T22:59:26.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.949+0000 7f4ef2982700 1 -- 172.21.15.153:0/26375622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4eec1027e0 msgr2=0x7f4eec0fbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:26.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.949+0000 7f4ef2982700 1 --2- 172.21.15.153:0/26375622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4eec1027e0 0x7f4eec0fbea0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0004940 tx=0x7f4ee0009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:26.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.949+0000 7f4ef2982700 1 -- 172.21.15.153:0/26375622 shutdown_connections 2024-03-11T22:59:26.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.949+0000 7f4ef2982700 1 --2- 172.21.15.153:0/26375622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4eec1027e0 0x7f4eec0fbea0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:26.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.949+0000 7f4ef2982700 1 --2- 172.21.15.153:0/26375622 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4eec101e40 0x7f4eec102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:26.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.949+0000 7f4ef2982700 1 -- 172.21.15.153:0/26375622 >> 172.21.15.153:0/26375622 conn(0x7f4eec071d90 msgr2=0x7f4eec072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:26.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.949+0000 7f4ef2982700 1 -- 172.21.15.153:0/26375622 shutdown_connections 2024-03-11T22:59:26.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.949+0000 7f4ef2982700 1 -- 172.21.15.153:0/26375622 wait complete. 2024-03-11T22:59:26.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.950+0000 7f4ef2982700 1 Processor -- start 2024-03-11T22:59:26.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.950+0000 7f4ef2982700 1 -- start start 2024-03-11T22:59:26.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.950+0000 7f4ef2982700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4eec101e40 0x7f4eec10eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:26.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.950+0000 7f4ef2982700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4eec1027e0 0x7f4eec10f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:26.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.950+0000 7f4ef2982700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eec10f680 con 0x7f4eec1027e0 2024-03-11T22:59:26.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.950+0000 7f4ef2982700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eec10f7f0 con 0x7f4eec101e40 2024-03-11T22:59:26.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.950+0000 7f4eeb7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4eec1027e0 0x7f4eec10f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:26.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.950+0000 7f4eeb7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4eec1027e0 0x7f4eec10f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:45160/0 (socket says 172.21.15.153:45160) 2024-03-11T22:59:26.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.950+0000 7f4eeb7fe700 1 -- 172.21.15.153:0/2881909208 learned_addr learned my addr 172.21.15.153:0/2881909208 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:59:26.971 INFO:teuthology.orchestra.run.smithi153.stdout:115964117014 2024-03-11T22:59:26.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.950+0000 7f4eebfff700 1 --2- 172.21.15.153:0/2881909208 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4eec101e40 0x7f4eec10eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:26.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.950+0000 7f4eeb7fe700 1 -- 172.21.15.153:0/2881909208 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4eec101e40 msgr2=0x7f4eec10eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:26.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.950+0000 7f4eeb7fe700 1 --2- 172.21.15.153:0/2881909208 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4eec101e40 0x7f4eec10eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:26.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.950+0000 7f4eeb7fe700 1 -- 172.21.15.153:0/2881909208 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee0005040 con 0x7f4eec1027e0 2024-03-11T22:59:26.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.950+0000 7f4eebfff700 1 --2- 172.21.15.153:0/2881909208 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4eec101e40 0x7f4eec10eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T22:59:26.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.951+0000 7f4eeb7fe700 1 --2- 172.21.15.153:0/2881909208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4eec1027e0 0x7f4eec10f030 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0009100 tx=0x7f4ee0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:26.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.951+0000 7f4ee8ff9700 1 -- 172.21.15.153:0/2881909208 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ee000e040 con 0x7f4eec1027e0 2024-03-11T22:59:26.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.951+0000 7f4ef2982700 1 -- 172.21.15.153:0/2881909208 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eec117a60 con 0x7f4eec1027e0 2024-03-11T22:59:26.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.951+0000 7f4ee8ff9700 1 -- 172.21.15.153:0/2881909208 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4ee000bbf0 con 0x7f4eec1027e0 2024-03-11T22:59:26.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.951+0000 7f4ee8ff9700 1 -- 172.21.15.153:0/2881909208 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ee001c630 con 0x7f4eec1027e0 2024-03-11T22:59:26.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.951+0000 7f4ef2982700 1 -- 172.21.15.153:0/2881909208 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eec117fa0 con 0x7f4eec1027e0 2024-03-11T22:59:26.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.952+0000 7f4ef2982700 1 -- 172.21.15.153:0/2881909208 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eec104310 con 0x7f4eec1027e0 2024-03-11T22:59:26.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.952+0000 7f4ee8ff9700 1 -- 172.21.15.153:0/2881909208 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f4ee000bd60 con 0x7f4eec1027e0 2024-03-11T22:59:26.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.953+0000 7f4ee8ff9700 1 --2- 172.21.15.153:0/2881909208 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f4ed40692a0 0x7f4ed406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:26.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.953+0000 7f4ee8ff9700 1 -- 172.21.15.153:0/2881909208 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4ee00881e0 con 0x7f4eec1027e0 2024-03-11T22:59:26.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.953+0000 7f4eebfff700 1 --2- 172.21.15.153:0/2881909208 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f4ed40692a0 0x7f4ed406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:26.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.953+0000 7f4eebfff700 1 --2- 172.21.15.153:0/2881909208 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f4ed40692a0 0x7f4ed406b750 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4edc009fd0 tx=0x7f4edc009e50 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:26.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:25.957+0000 7f4ee8ff9700 1 -- 172.21.15.153:0/2881909208 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f4ee0054850 con 0x7f4eec1027e0 2024-03-11T22:59:26.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:26.145+0000 7f4ef2982700 1 -- 172.21.15.153:0/2881909208 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f4eec04bf00 con 0x7f4eec1027e0 2024-03-11T22:59:26.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:26.145+0000 7f4ee8ff9700 1 -- 172.21.15.153:0/2881909208 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f4ee0057e70 con 0x7f4eec1027e0 2024-03-11T22:59:26.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:26.147+0000 7f4ef2982700 1 -- 172.21.15.153:0/2881909208 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f4ed40692a0 msgr2=0x7f4ed406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:26.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:26.147+0000 7f4ef2982700 1 --2- 172.21.15.153:0/2881909208 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f4ed40692a0 0x7f4ed406b750 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4edc009fd0 tx=0x7f4edc009e50 comp rx=0 tx=0).stop 2024-03-11T22:59:26.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:26.147+0000 7f4ef2982700 1 -- 172.21.15.153:0/2881909208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4eec1027e0 msgr2=0x7f4eec10f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:26.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:26.147+0000 7f4ef2982700 1 --2- 172.21.15.153:0/2881909208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4eec1027e0 0x7f4eec10f030 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4ee0009100 tx=0x7f4ee0009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:26.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:26.147+0000 7f4ef2982700 1 -- 172.21.15.153:0/2881909208 shutdown_connections 2024-03-11T22:59:26.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:26.147+0000 7f4ef2982700 1 --2- 172.21.15.153:0/2881909208 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f4ed40692a0 0x7f4ed406b750 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:26.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:26.147+0000 7f4ef2982700 1 --2- 172.21.15.153:0/2881909208 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4eec101e40 0x7f4eec10eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:26.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:26.147+0000 7f4ef2982700 1 --2- 172.21.15.153:0/2881909208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4eec1027e0 0x7f4eec10f030 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:26.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:26.147+0000 7f4ef2982700 1 -- 172.21.15.153:0/2881909208 >> 172.21.15.153:0/2881909208 conn(0x7f4eec071d90 msgr2=0x7f4eec0fae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:26.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:26.147+0000 7f4ef2982700 1 -- 172.21.15.153:0/2881909208 shutdown_connections 2024-03-11T22:59:26.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:26.147+0000 7f4ef2982700 1 -- 172.21.15.153:0/2881909208 wait complete. 2024-03-11T22:59:27.525 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:27 smithi153 ceph-mon[28962]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:27.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.619+0000 7f2f7e504700 1 -- 172.21.15.153:0/3898496446 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f781038f0 msgr2=0x7f2f780fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.619+0000 7f2f7e504700 1 --2- 172.21.15.153:0/3898496446 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f781038f0 0x7f2f780fcf60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2f68004940 tx=0x7f2f68009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.619+0000 7f2f7e504700 1 -- 172.21.15.153:0/3898496446 shutdown_connections 2024-03-11T22:59:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.619+0000 7f2f7e504700 1 --2- 172.21.15.153:0/3898496446 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f781038f0 0x7f2f780fcf60 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.619+0000 7f2f7e504700 1 --2- 172.21.15.153:0/3898496446 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2f78102f50 0x7f2f78103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.619+0000 7f2f7e504700 1 -- 172.21.15.153:0/3898496446 >> 172.21.15.153:0/3898496446 conn(0x7f2f780f8ae0 msgr2=0x7f2f780faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.619+0000 7f2f7e504700 1 -- 172.21.15.153:0/3898496446 shutdown_connections 2024-03-11T22:59:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.619+0000 7f2f7e504700 1 -- 172.21.15.153:0/3898496446 wait complete. 2024-03-11T22:59:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.619+0000 7f2f7e504700 1 Processor -- start 2024-03-11T22:59:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.620+0000 7f2f7e504700 1 -- start start 2024-03-11T22:59:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.620+0000 7f2f7e504700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f78102f50 0x7f2f78070290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.620+0000 7f2f7e504700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2f781038f0 0x7f2f780707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.620+0000 7f2f7e504700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f78070dd0 con 0x7f2f78102f50 2024-03-11T22:59:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.620+0000 7f2f7e504700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f78070f40 con 0x7f2f781038f0 2024-03-11T22:59:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.620+0000 7f2f77fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f78102f50 0x7f2f78070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:27.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.620+0000 7f2f77fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f78102f50 0x7f2f78070290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:45176/0 (socket says 172.21.15.153:45176) 2024-03-11T22:59:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.620+0000 7f2f77fff700 1 -- 172.21.15.153:0/1299380904 learned_addr learned my addr 172.21.15.153:0/1299380904 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:59:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.620+0000 7f2f777fe700 1 --2- 172.21.15.153:0/1299380904 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2f781038f0 0x7f2f780707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.620+0000 7f2f77fff700 1 -- 172.21.15.153:0/1299380904 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2f781038f0 msgr2=0x7f2f780707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.620+0000 7f2f77fff700 1 --2- 172.21.15.153:0/1299380904 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2f781038f0 0x7f2f780707d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.620+0000 7f2f77fff700 1 -- 172.21.15.153:0/1299380904 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f68005040 con 0x7f2f78102f50 2024-03-11T22:59:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.620+0000 7f2f77fff700 1 --2- 172.21.15.153:0/1299380904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f78102f50 0x7f2f78070290 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c00c830 tx=0x7f2f6c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.621+0000 7f2f74ff9700 1 -- 172.21.15.153:0/1299380904 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f6c00a760 con 0x7f2f78102f50 2024-03-11T22:59:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.621+0000 7f2f7e504700 1 -- 172.21.15.153:0/1299380904 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f78119cd0 con 0x7f2f78102f50 2024-03-11T22:59:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.621+0000 7f2f74ff9700 1 -- 172.21.15.153:0/1299380904 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2f6c00ad90 con 0x7f2f78102f50 2024-03-11T22:59:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.621+0000 7f2f74ff9700 1 -- 172.21.15.153:0/1299380904 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f6c0113a0 con 0x7f2f78102f50 2024-03-11T22:59:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.621+0000 7f2f7e504700 1 -- 172.21.15.153:0/1299380904 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f7811a1b0 con 0x7f2f78102f50 2024-03-11T22:59:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.622+0000 7f2f7e504700 1 -- 172.21.15.153:0/1299380904 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f78105420 con 0x7f2f78102f50 2024-03-11T22:59:27.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.622+0000 7f2f74ff9700 1 -- 172.21.15.153:0/1299380904 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f2f6c012400 con 0x7f2f78102f50 2024-03-11T22:59:27.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.622+0000 7f2f74ff9700 1 --2- 172.21.15.153:0/1299380904 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f2f600692a0 0x7f2f6006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:27.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.623+0000 7f2f777fe700 1 --2- 172.21.15.153:0/1299380904 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f2f600692a0 0x7f2f6006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:27.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.623+0000 7f2f74ff9700 1 -- 172.21.15.153:0/1299380904 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2f6c086eb0 con 0x7f2f78102f50 2024-03-11T22:59:27.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.623+0000 7f2f777fe700 1 --2- 172.21.15.153:0/1299380904 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f2f600692a0 0x7f2f6006b750 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2f6800b3b0 tx=0x7f2f68009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:27.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.627+0000 7f2f74ff9700 1 -- 172.21.15.153:0/1299380904 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f2f6c053520 con 0x7f2f78102f50 2024-03-11T22:59:27.639 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:27 smithi163 ceph-mon[35584]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:27.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.816+0000 7f2f7e504700 1 -- 172.21.15.153:0/1299380904 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f2f780719d0 con 0x7f2f78102f50 2024-03-11T22:59:27.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.816+0000 7f2f74ff9700 1 -- 172.21.15.153:0/1299380904 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2f6c05c020 con 0x7f2f78102f50 2024-03-11T22:59:27.821 INFO:teuthology.orchestra.run.smithi153.stdout:81604378652 2024-03-11T22:59:27.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.818+0000 7f2f7e504700 1 -- 172.21.15.153:0/1299380904 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f2f600692a0 msgr2=0x7f2f6006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:27.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.818+0000 7f2f7e504700 1 --2- 172.21.15.153:0/1299380904 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f2f600692a0 0x7f2f6006b750 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2f6800b3b0 tx=0x7f2f68009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:27.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.818+0000 7f2f7e504700 1 -- 172.21.15.153:0/1299380904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f78102f50 msgr2=0x7f2f78070290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:27.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.818+0000 7f2f7e504700 1 --2- 172.21.15.153:0/1299380904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f78102f50 0x7f2f78070290 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c00c830 tx=0x7f2f6c004470 comp rx=0 tx=0).stop 2024-03-11T22:59:27.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.818+0000 7f2f7e504700 1 -- 172.21.15.153:0/1299380904 shutdown_connections 2024-03-11T22:59:27.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.818+0000 7f2f7e504700 1 --2- 172.21.15.153:0/1299380904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2f78102f50 0x7f2f78070290 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:27.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.818+0000 7f2f7e504700 1 --2- 172.21.15.153:0/1299380904 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f2f600692a0 0x7f2f6006b750 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:27.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.818+0000 7f2f7e504700 1 --2- 172.21.15.153:0/1299380904 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2f781038f0 0x7f2f780707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:27.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.818+0000 7f2f7e504700 1 -- 172.21.15.153:0/1299380904 >> 172.21.15.153:0/1299380904 conn(0x7f2f780f8ae0 msgr2=0x7f2f780fa5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:27.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.818+0000 7f2f7e504700 1 -- 172.21.15.153:0/1299380904 shutdown_connections 2024-03-11T22:59:27.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:27.818+0000 7f2f7e504700 1 -- 172.21.15.153:0/1299380904 wait complete. 2024-03-11T22:59:28.145 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117012 got 115964117014 for osd.3 2024-03-11T22:59:28.145 DEBUG:teuthology.parallel:result is None 2024-03-11T22:59:28.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:28 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/1299380904' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T22:59:28.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.542+0000 7f77da7b2700 1 -- 172.21.15.153:0/510760535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77d4101e40 msgr2=0x7f77d4102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:28.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.542+0000 7f77da7b2700 1 --2- 172.21.15.153:0/510760535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77d4101e40 0x7f77d4102210 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f77c4004940 tx=0x7f77c4009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:28.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.542+0000 7f77da7b2700 1 -- 172.21.15.153:0/510760535 shutdown_connections 2024-03-11T22:59:28.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.542+0000 7f77da7b2700 1 --2- 172.21.15.153:0/510760535 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f77d41027e0 0x7f77d40fbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:28.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.542+0000 7f77da7b2700 1 --2- 172.21.15.153:0/510760535 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77d4101e40 0x7f77d4102210 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:28.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.542+0000 7f77da7b2700 1 -- 172.21.15.153:0/510760535 >> 172.21.15.153:0/510760535 conn(0x7f77d4071d90 msgr2=0x7f77d4072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:28.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.542+0000 7f77da7b2700 1 -- 172.21.15.153:0/510760535 shutdown_connections 2024-03-11T22:59:28.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.542+0000 7f77da7b2700 1 -- 172.21.15.153:0/510760535 wait complete. 2024-03-11T22:59:28.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.542+0000 7f77da7b2700 1 Processor -- start 2024-03-11T22:59:28.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.543+0000 7f77da7b2700 1 -- start start 2024-03-11T22:59:28.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.543+0000 7f77da7b2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77d4101e40 0x7f77d4113a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:28.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.543+0000 7f77da7b2700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f77d41027e0 0x7f77d4113f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:28.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.543+0000 7f77da7b2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d4114660 con 0x7f77d4101e40 2024-03-11T22:59:28.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.543+0000 7f77da7b2700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d410d900 con 0x7f77d41027e0 2024-03-11T22:59:28.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.543+0000 7f77d3fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77d4101e40 0x7f77d4113a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:28.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.543+0000 7f77d3fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77d4101e40 0x7f77d4113a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:45188/0 (socket says 172.21.15.153:45188) 2024-03-11T22:59:28.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.543+0000 7f77d3fff700 1 -- 172.21.15.153:0/170833640 learned_addr learned my addr 172.21.15.153:0/170833640 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:59:28.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.543+0000 7f77d37fe700 1 --2- 172.21.15.153:0/170833640 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f77d41027e0 0x7f77d4113f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:28.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.543+0000 7f77d3fff700 1 -- 172.21.15.153:0/170833640 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f77d41027e0 msgr2=0x7f77d4113f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:28.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.543+0000 7f77d3fff700 1 --2- 172.21.15.153:0/170833640 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f77d41027e0 0x7f77d4113f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:28.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.543+0000 7f77d3fff700 1 -- 172.21.15.153:0/170833640 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c4005040 con 0x7f77d4101e40 2024-03-11T22:59:28.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.543+0000 7f77d37fe700 1 --2- 172.21.15.153:0/170833640 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f77d41027e0 0x7f77d4113f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T22:59:28.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.543+0000 7f77d3fff700 1 --2- 172.21.15.153:0/170833640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77d4101e40 0x7f77d4113a40 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f77c400b3b0 tx=0x7f77c4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:28.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.543+0000 7f77d0ff9700 1 -- 172.21.15.153:0/170833640 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77c400e040 con 0x7f77d4101e40 2024-03-11T22:59:28.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.544+0000 7f77d0ff9700 1 -- 172.21.15.153:0/170833640 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f77c400bbf0 con 0x7f77d4101e40 2024-03-11T22:59:28.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.544+0000 7f77d0ff9700 1 -- 172.21.15.153:0/170833640 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f77c401c630 con 0x7f77d4101e40 2024-03-11T22:59:28.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.544+0000 7f77da7b2700 1 -- 172.21.15.153:0/170833640 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77d410db80 con 0x7f77d4101e40 2024-03-11T22:59:28.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.544+0000 7f77da7b2700 1 -- 172.21.15.153:0/170833640 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77d410e0c0 con 0x7f77d4101e40 2024-03-11T22:59:28.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.544+0000 7f77da7b2700 1 -- 172.21.15.153:0/170833640 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77d40738b0 con 0x7f77d4101e40 2024-03-11T22:59:28.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.545+0000 7f77d0ff9700 1 -- 172.21.15.153:0/170833640 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f77c400bd60 con 0x7f77d4101e40 2024-03-11T22:59:28.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.545+0000 7f77d0ff9700 1 --2- 172.21.15.153:0/170833640 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f77bc0692a0 0x7f77bc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:28.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.545+0000 7f77d0ff9700 1 -- 172.21.15.153:0/170833640 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f77c4088170 con 0x7f77d4101e40 2024-03-11T22:59:28.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.545+0000 7f77d37fe700 1 --2- 172.21.15.153:0/170833640 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f77bc0692a0 0x7f77bc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:28.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.546+0000 7f77d37fe700 1 --2- 172.21.15.153:0/170833640 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f77bc0692a0 0x7f77bc06b750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f77c8006fd0 tx=0x7f77c8008040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:28.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.550+0000 7f77d0ff9700 1 -- 172.21.15.153:0/170833640 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f77c4058310 con 0x7f77d4101e40 2024-03-11T22:59:28.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.737+0000 7f77da7b2700 1 -- 172.21.15.153:0/170833640 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f77d410f050 con 0x7f77d4101e40 2024-03-11T22:59:28.742 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:28 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/1299380904' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T22:59:28.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.737+0000 7f77d0ff9700 1 -- 172.21.15.153:0/170833640 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f77c4057ea0 con 0x7f77d4101e40 2024-03-11T22:59:28.742 INFO:teuthology.orchestra.run.smithi153.stdout:60129542177 2024-03-11T22:59:28.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.740+0000 7f77da7b2700 1 -- 172.21.15.153:0/170833640 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f77bc0692a0 msgr2=0x7f77bc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:28.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.740+0000 7f77da7b2700 1 --2- 172.21.15.153:0/170833640 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f77bc0692a0 0x7f77bc06b750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f77c8006fd0 tx=0x7f77c8008040 comp rx=0 tx=0).stop 2024-03-11T22:59:28.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.740+0000 7f77da7b2700 1 -- 172.21.15.153:0/170833640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77d4101e40 msgr2=0x7f77d4113a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:28.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.740+0000 7f77da7b2700 1 --2- 172.21.15.153:0/170833640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77d4101e40 0x7f77d4113a40 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f77c400b3b0 tx=0x7f77c4009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:28.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.740+0000 7f77da7b2700 1 -- 172.21.15.153:0/170833640 shutdown_connections 2024-03-11T22:59:28.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.740+0000 7f77da7b2700 1 --2- 172.21.15.153:0/170833640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77d4101e40 0x7f77d4113a40 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:28.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.740+0000 7f77da7b2700 1 --2- 172.21.15.153:0/170833640 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f77bc0692a0 0x7f77bc06b750 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:28.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.740+0000 7f77da7b2700 1 --2- 172.21.15.153:0/170833640 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f77d41027e0 0x7f77d4113f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:28.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.740+0000 7f77da7b2700 1 -- 172.21.15.153:0/170833640 >> 172.21.15.153:0/170833640 conn(0x7f77d4071d90 msgr2=0x7f77d4105250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:28.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.740+0000 7f77da7b2700 1 -- 172.21.15.153:0/170833640 shutdown_connections 2024-03-11T22:59:28.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:28.740+0000 7f77da7b2700 1 -- 172.21.15.153:0/170833640 wait complete. 2024-03-11T22:59:29.384 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691848 got 171798691851 for osd.5 2024-03-11T22:59:29.384 DEBUG:teuthology.parallel:result is None 2024-03-11T22:59:29.656 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:29 smithi153 ceph-mon[28962]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:29.656 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:29 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/170833640' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T22:59:29.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:29 smithi163 ceph-mon[35584]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:29.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:29 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/170833640' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T22:59:29.948 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378649 got 81604378652 for osd.2 2024-03-11T22:59:29.948 DEBUG:teuthology.parallel:result is None 2024-03-11T22:59:30.513 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542175 got 60129542177 for osd.1 2024-03-11T22:59:30.513 DEBUG:teuthology.parallel:result is None 2024-03-11T22:59:30.513 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-11T22:59:30.513 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:59:30.775 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:59:31.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:31 smithi153 ceph-mon[28962]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:31.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:31 smithi163 ceph-mon[35584]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:32.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.370+0000 7f6b9e85c700 1 -- 172.21.15.153:0/1898973475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b98102f50 msgr2=0x7f6b98103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:32.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.370+0000 7f6b9e85c700 1 --2- 172.21.15.153:0/1898973475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b98102f50 0x7f6b98103320 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c004940 tx=0x7f6b8c009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:32.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.370+0000 7f6b9e85c700 1 -- 172.21.15.153:0/1898973475 shutdown_connections 2024-03-11T22:59:32.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.370+0000 7f6b9e85c700 1 --2- 172.21.15.153:0/1898973475 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b981038f0 0x7f6b980fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:32.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.370+0000 7f6b9e85c700 1 --2- 172.21.15.153:0/1898973475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b98102f50 0x7f6b98103320 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:32.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.370+0000 7f6b9e85c700 1 -- 172.21.15.153:0/1898973475 >> 172.21.15.153:0/1898973475 conn(0x7f6b980f8ae0 msgr2=0x7f6b980faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:32.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.370+0000 7f6b9e85c700 1 -- 172.21.15.153:0/1898973475 shutdown_connections 2024-03-11T22:59:32.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.370+0000 7f6b9e85c700 1 -- 172.21.15.153:0/1898973475 wait complete. 2024-03-11T22:59:32.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.371+0000 7f6b9e85c700 1 Processor -- start 2024-03-11T22:59:32.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.371+0000 7f6b9e85c700 1 -- start start 2024-03-11T22:59:32.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.371+0000 7f6b9e85c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b98102f50 0x7f6b98113a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:32.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.371+0000 7f6b9e85c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b981038f0 0x7f6b98113f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:32.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.371+0000 7f6b9e85c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b981145e0 con 0x7f6b98102f50 2024-03-11T22:59:32.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.371+0000 7f6b9e85c700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9810d910 con 0x7f6b981038f0 2024-03-11T22:59:32.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.372+0000 7f6b97fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b98102f50 0x7f6b98113a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:32.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.372+0000 7f6b97fff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b98102f50 0x7f6b98113a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49994/0 (socket says 172.21.15.153:49994) 2024-03-11T22:59:32.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.372+0000 7f6b97fff700 1 -- 172.21.15.153:0/3494374280 learned_addr learned my addr 172.21.15.153:0/3494374280 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:59:32.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.372+0000 7f6b977fe700 1 --2- 172.21.15.153:0/3494374280 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b981038f0 0x7f6b98113f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:32.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.372+0000 7f6b97fff700 1 -- 172.21.15.153:0/3494374280 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b981038f0 msgr2=0x7f6b98113f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:32.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.372+0000 7f6b97fff700 1 --2- 172.21.15.153:0/3494374280 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b981038f0 0x7f6b98113f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:32.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.372+0000 7f6b97fff700 1 -- 172.21.15.153:0/3494374280 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b8c005040 con 0x7f6b98102f50 2024-03-11T22:59:32.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.372+0000 7f6b97fff700 1 --2- 172.21.15.153:0/3494374280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b98102f50 0x7f6b98113a50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c009fd0 tx=0x7f6b8c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:32.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.372+0000 7f6b94ff9700 1 -- 172.21.15.153:0/3494374280 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b8c00e040 con 0x7f6b98102f50 2024-03-11T22:59:32.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.372+0000 7f6b9e85c700 1 -- 172.21.15.153:0/3494374280 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b9810db90 con 0x7f6b98102f50 2024-03-11T22:59:32.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.372+0000 7f6b94ff9700 1 -- 172.21.15.153:0/3494374280 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6b8c00bbf0 con 0x7f6b98102f50 2024-03-11T22:59:32.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.372+0000 7f6b94ff9700 1 -- 172.21.15.153:0/3494374280 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6b8c01c630 con 0x7f6b98102f50 2024-03-11T22:59:32.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.372+0000 7f6b9e85c700 1 -- 172.21.15.153:0/3494374280 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b9810e0d0 con 0x7f6b98102f50 2024-03-11T22:59:32.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.373+0000 7f6b9e85c700 1 -- 172.21.15.153:0/3494374280 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b980fc6e0 con 0x7f6b98102f50 2024-03-11T22:59:32.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.374+0000 7f6b94ff9700 1 -- 172.21.15.153:0/3494374280 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f6b8c00bd60 con 0x7f6b98102f50 2024-03-11T22:59:32.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.374+0000 7f6b94ff9700 1 --2- 172.21.15.153:0/3494374280 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f6b800692b0 0x7f6b8006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:32.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.374+0000 7f6b94ff9700 1 -- 172.21.15.153:0/3494374280 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6b8c088080 con 0x7f6b98102f50 2024-03-11T22:59:32.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.374+0000 7f6b977fe700 1 --2- 172.21.15.153:0/3494374280 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f6b800692b0 0x7f6b8006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:32.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.375+0000 7f6b977fe700 1 --2- 172.21.15.153:0/3494374280 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f6b800692b0 0x7f6b8006b760 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6b9810f280 tx=0x7f6b8800b040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:32.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.379+0000 7f6b94ff9700 1 -- 172.21.15.153:0/3494374280 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f6b8c058180 con 0x7f6b98102f50 2024-03-11T22:59:32.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.557+0000 7f6b9e85c700 1 -- 172.21.15.153:0/3494374280 --> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6b9810f0d0 con 0x7f6b800692b0 2024-03-11T22:59:32.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.559+0000 7f6b94ff9700 1 -- 172.21.15.153:0/3494374280 <== mgr.14204 v2:172.21.15.153:6800/29938766 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28301 (secure 0 0 0) 0x7f6b9810f0d0 con 0x7f6b800692b0 2024-03-11T22:59:32.564 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:59:32.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.561+0000 7f6b9e85c700 1 -- 172.21.15.153:0/3494374280 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f6b800692b0 msgr2=0x7f6b8006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:32.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.561+0000 7f6b9e85c700 1 --2- 172.21.15.153:0/3494374280 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f6b800692b0 0x7f6b8006b760 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6b9810f280 tx=0x7f6b8800b040 comp rx=0 tx=0).stop 2024-03-11T22:59:32.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.561+0000 7f6b9e85c700 1 -- 172.21.15.153:0/3494374280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b98102f50 msgr2=0x7f6b98113a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:32.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.561+0000 7f6b9e85c700 1 --2- 172.21.15.153:0/3494374280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b98102f50 0x7f6b98113a50 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6b8c009fd0 tx=0x7f6b8c009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:32.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.561+0000 7f6b9e85c700 1 -- 172.21.15.153:0/3494374280 shutdown_connections 2024-03-11T22:59:32.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.561+0000 7f6b9e85c700 1 --2- 172.21.15.153:0/3494374280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6b98102f50 0x7f6b98113a50 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:32.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.561+0000 7f6b9e85c700 1 --2- 172.21.15.153:0/3494374280 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f6b800692b0 0x7f6b8006b760 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:32.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.561+0000 7f6b9e85c700 1 --2- 172.21.15.153:0/3494374280 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b981038f0 0x7f6b98113f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:32.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.561+0000 7f6b9e85c700 1 -- 172.21.15.153:0/3494374280 >> 172.21.15.153:0/3494374280 conn(0x7f6b980f8ae0 msgr2=0x7f6b980fa500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:32.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.561+0000 7f6b9e85c700 1 -- 172.21.15.153:0/3494374280 shutdown_connections 2024-03-11T22:59:32.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:32.561+0000 7f6b9e85c700 1 -- 172.21.15.153:0/3494374280 wait complete. 2024-03-11T22:59:32.566 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-03-11T22:59:33.148 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2024-03-11T22:59:32.089952+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787652,"kb_used_data":3924,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683260,"statfs":{"total":575970213888,"available":574139658240,"internally_reserved":0,"allocated":4018176,"data_stored":2624466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002091"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":189,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-11T22:58:40.150235+0000","last_change":"2024-03-11T22:58:14.685011+0000","last_active":"2024-03-11T22:58:40.150235+0000","last_peered":"2024-03-11T22:58:40.150235+0000","last_clean":"2024-03-11T22:58:40.150235+0000","last_became_active":"2024-03-11T22:58:14.684689+0000","last_became_peered":"2024-03-11T22:58:14.684689+0000","last_unstale":"2024-03-11T22:58:40.150235+0000","last_undegraded":"2024-03-11T22:58:40.150235+0000","last_fullsized":"2024-03-11T22:58:40.150235+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:26.418919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:26.418919+0000","last_clean_scrub_stamp":"2024-03-11T22:57:26.418919+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:35:12.632149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]}]},{"osd":4,"up_from":35,"seq":150323855379,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 11 22:59:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.47099999999999997}]},{"osd":1,"last update":"Mon Mar 11 22:59:17 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.38700000000000001}]},{"osd":2,"last update":"Mon Mar 11 22:59:17 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.44},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.52100000000000002}]},{"osd":3,"last update":"Mon Mar 11 22:59:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.35999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]}]},{"osd":3,"up_from":27,"seq":115964117015,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 11 22:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.442},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.39200000000000002}]},{"osd":1,"last update":"Mon Mar 11 22:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.34300000000000003}]},{"osd":2,"last update":"Mon Mar 11 22:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.41999999999999998}]},{"osd":4,"last update":"Mon Mar 11 22:59:17 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.51900000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 11 22:59:27 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.37,"15min":0.38},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.39700000000000002,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.376,"15min":0.38400000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.41099999999999998,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.29199999999999998}]},{"osd":1,"last update":"Mon Mar 11 22:59:27 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.38900000000000001,"15min":0.39700000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.47599999999999998,"5min":0.53100000000000003,"15min":0.53100000000000003},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.39600000000000002,"15min":0.40400000000000003},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.46100000000000002,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.307}]},{"osd":3,"last update":"Mon Mar 11 22:58:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.59799999999999998}]},{"osd":4,"last update":"Mon Mar 11 22:59:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.55800000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 11 22:59:01 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.32800000000000001,"15min":0.32300000000000001},"min":{"1min":0.26200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.35199999999999998,"15min":0.34399999999999997},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.25700000000000001}]},{"osd":2,"last update":"Mon Mar 11 22:59:28 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.39300000000000002,"15min":0.39100000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.308},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.38400000000000001,"15min":0.38200000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.23300000000000001}]},{"osd":3,"last update":"Mon Mar 11 22:58:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.622},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.55700000000000005}]},{"osd":4,"last update":"Mon Mar 11 22:59:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.57199999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]}]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 11 22:59:02 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.33000000000000002,"15min":0.32600000000000001},"min":{"1min":0.251,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.35699999999999998,"15min":0.35899999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.495,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.311}]},{"osd":2,"last update":"Mon Mar 11 22:58:29 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.248},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.29699999999999999}]},{"osd":3,"last update":"Mon Mar 11 22:58:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.64300000000000002}]},{"osd":4,"last update":"Mon Mar 11 22:59:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.62}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:59:33.149 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:59:33.403 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:59:33.525 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:33 smithi153 ceph-mon[28962]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:33.525 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:33 smithi153 ceph-mon[28962]: from='client.14404 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:59:33.526 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:33 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:59:33.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:33 smithi163 ceph-mon[35584]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:33.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:33 smithi163 ceph-mon[35584]: from='client.14404 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:59:33.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:33 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:59:35.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.138+0000 7f5dee178700 1 -- 172.21.15.153:0/826415180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5de8103140 msgr2=0x7f5de8103510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:35.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.138+0000 7f5dee178700 1 --2- 172.21.15.153:0/826415180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5de8103140 0x7f5de8103510 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5dd8004940 tx=0x7f5dd8009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:35.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.138+0000 7f5dee178700 1 -- 172.21.15.153:0/826415180 shutdown_connections 2024-03-11T22:59:35.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.138+0000 7f5dee178700 1 --2- 172.21.15.153:0/826415180 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5de80fb570 0x7f5de80fb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:35.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.138+0000 7f5dee178700 1 --2- 172.21.15.153:0/826415180 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5de8103140 0x7f5de8103510 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:35.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.138+0000 7f5dee178700 1 -- 172.21.15.153:0/826415180 >> 172.21.15.153:0/826415180 conn(0x7f5de8071560 msgr2=0x7f5de8071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:35.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.138+0000 7f5dee178700 1 -- 172.21.15.153:0/826415180 shutdown_connections 2024-03-11T22:59:35.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.138+0000 7f5dee178700 1 -- 172.21.15.153:0/826415180 wait complete. 2024-03-11T22:59:35.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.139+0000 7f5dee178700 1 Processor -- start 2024-03-11T22:59:35.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.139+0000 7f5dee178700 1 -- start start 2024-03-11T22:59:35.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.139+0000 7f5dee178700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5de80fb570 0x7f5de8113bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:35.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.139+0000 7f5dee178700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5de8103140 0x7f5de8114130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:35.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.139+0000 7f5dee178700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5de8114810 con 0x7f5de80fb570 2024-03-11T22:59:35.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.139+0000 7f5dee178700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5de810dab0 con 0x7f5de8103140 2024-03-11T22:59:35.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.139+0000 7f5de77fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5de80fb570 0x7f5de8113bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:35.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.139+0000 7f5de77fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5de80fb570 0x7f5de8113bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50006/0 (socket says 172.21.15.153:50006) 2024-03-11T22:59:35.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.139+0000 7f5de77fe700 1 -- 172.21.15.153:0/1706633516 learned_addr learned my addr 172.21.15.153:0/1706633516 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:59:35.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.139+0000 7f5de6ffd700 1 --2- 172.21.15.153:0/1706633516 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5de8103140 0x7f5de8114130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:35.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.140+0000 7f5de77fe700 1 -- 172.21.15.153:0/1706633516 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5de8103140 msgr2=0x7f5de8114130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:35.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.140+0000 7f5de77fe700 1 --2- 172.21.15.153:0/1706633516 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5de8103140 0x7f5de8114130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:35.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.140+0000 7f5de77fe700 1 -- 172.21.15.153:0/1706633516 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd8005040 con 0x7f5de80fb570 2024-03-11T22:59:35.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.140+0000 7f5de77fe700 1 --2- 172.21.15.153:0/1706633516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5de80fb570 0x7f5de8113bf0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5dd800b3b0 tx=0x7f5dd8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:35.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.140+0000 7f5dcffff700 1 -- 172.21.15.153:0/1706633516 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5dd800e040 con 0x7f5de80fb570 2024-03-11T22:59:35.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.140+0000 7f5dee178700 1 -- 172.21.15.153:0/1706633516 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5de810dd30 con 0x7f5de80fb570 2024-03-11T22:59:35.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.140+0000 7f5dcffff700 1 -- 172.21.15.153:0/1706633516 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5dd800bbf0 con 0x7f5de80fb570 2024-03-11T22:59:35.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.140+0000 7f5dcffff700 1 -- 172.21.15.153:0/1706633516 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5dd801c630 con 0x7f5de80fb570 2024-03-11T22:59:35.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.140+0000 7f5dee178700 1 -- 172.21.15.153:0/1706633516 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5de810e270 con 0x7f5de80fb570 2024-03-11T22:59:35.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.141+0000 7f5dee178700 1 -- 172.21.15.153:0/1706633516 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5de8105570 con 0x7f5de80fb570 2024-03-11T22:59:35.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.141+0000 7f5dcffff700 1 -- 172.21.15.153:0/1706633516 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f5dd801c8f0 con 0x7f5de80fb570 2024-03-11T22:59:35.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.142+0000 7f5dcffff700 1 --2- 172.21.15.153:0/1706633516 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f5dd00692a0 0x7f5dd006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:35.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.142+0000 7f5dcffff700 1 -- 172.21.15.153:0/1706633516 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5dd8088240 con 0x7f5de80fb570 2024-03-11T22:59:35.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.142+0000 7f5de6ffd700 1 --2- 172.21.15.153:0/1706633516 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f5dd00692a0 0x7f5dd006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:35.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.142+0000 7f5de6ffd700 1 --2- 172.21.15.153:0/1706633516 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f5dd00692a0 0x7f5dd006b750 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc006fd0 tx=0x7f5ddc008040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:35.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.146+0000 7f5dcffff700 1 -- 172.21.15.153:0/1706633516 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f5dd80583b0 con 0x7f5de80fb570 2024-03-11T22:59:35.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.325+0000 7f5dee178700 1 -- 172.21.15.153:0/1706633516 --> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5de810a230 con 0x7f5dd00692a0 2024-03-11T22:59:35.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.327+0000 7f5dcffff700 1 -- 172.21.15.153:0/1706633516 <== mgr.14204 v2:172.21.15.153:6800/29938766 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28314 (secure 0 0 0) 0x7f5de810a230 con 0x7f5dd00692a0 2024-03-11T22:59:35.331 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:59:35.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.329+0000 7f5dee178700 1 -- 172.21.15.153:0/1706633516 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f5dd00692a0 msgr2=0x7f5dd006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:35.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.329+0000 7f5dee178700 1 --2- 172.21.15.153:0/1706633516 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f5dd00692a0 0x7f5dd006b750 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc006fd0 tx=0x7f5ddc008040 comp rx=0 tx=0).stop 2024-03-11T22:59:35.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.329+0000 7f5dee178700 1 -- 172.21.15.153:0/1706633516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5de80fb570 msgr2=0x7f5de8113bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:35.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.329+0000 7f5dee178700 1 --2- 172.21.15.153:0/1706633516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5de80fb570 0x7f5de8113bf0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5dd800b3b0 tx=0x7f5dd8009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:35.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.329+0000 7f5dee178700 1 -- 172.21.15.153:0/1706633516 shutdown_connections 2024-03-11T22:59:35.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.329+0000 7f5dee178700 1 --2- 172.21.15.153:0/1706633516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5de80fb570 0x7f5de8113bf0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:35.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.329+0000 7f5dee178700 1 --2- 172.21.15.153:0/1706633516 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f5dd00692a0 0x7f5dd006b750 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:35.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.329+0000 7f5dee178700 1 --2- 172.21.15.153:0/1706633516 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5de8103140 0x7f5de8114130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:35.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.329+0000 7f5dee178700 1 -- 172.21.15.153:0/1706633516 >> 172.21.15.153:0/1706633516 conn(0x7f5de8071560 msgr2=0x7f5de80f96e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:35.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.329+0000 7f5dee178700 1 -- 172.21.15.153:0/1706633516 shutdown_connections 2024-03-11T22:59:35.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:35.329+0000 7f5dee178700 1 -- 172.21.15.153:0/1706633516 wait complete. 2024-03-11T22:59:35.334 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-03-11T22:59:35.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:35 smithi153 ceph-mon[28962]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:35.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:35 smithi163 ceph-mon[35584]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:36.097 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":157,"stamp":"2024-03-11T22:59:34.090263+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787652,"kb_used_data":3924,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683260,"statfs":{"total":575970213888,"available":574139658240,"internally_reserved":0,"allocated":4018176,"data_stored":2624466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002110"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":189,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-11T22:58:40.150235+0000","last_change":"2024-03-11T22:58:14.685011+0000","last_active":"2024-03-11T22:58:40.150235+0000","last_peered":"2024-03-11T22:58:40.150235+0000","last_clean":"2024-03-11T22:58:40.150235+0000","last_became_active":"2024-03-11T22:58:14.684689+0000","last_became_peered":"2024-03-11T22:58:14.684689+0000","last_unstale":"2024-03-11T22:58:40.150235+0000","last_undegraded":"2024-03-11T22:58:40.150235+0000","last_fullsized":"2024-03-11T22:58:40.150235+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:26.418919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:26.418919+0000","last_clean_scrub_stamp":"2024-03-11T22:57:26.418919+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:35:12.632149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32900000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]}]},{"osd":4,"up_from":35,"seq":150323855379,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 11 22:59:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.625,"5min":0.625,"15min":0.625},"last":0.47099999999999997}]},{"osd":1,"last update":"Mon Mar 11 22:59:17 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.38700000000000001}]},{"osd":2,"last update":"Mon Mar 11 22:59:17 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.44},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.52100000000000002}]},{"osd":3,"last update":"Mon Mar 11 22:59:17 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.35999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]}]},{"osd":3,"up_from":27,"seq":115964117016,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 11 22:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.51400000000000001}]},{"osd":1,"last update":"Mon Mar 11 22:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.45700000000000002}]},{"osd":2,"last update":"Mon Mar 11 22:58:51 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.44600000000000001}]},{"osd":4,"last update":"Mon Mar 11 22:59:17 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.56599999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 11 22:59:27 2024","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.37,"15min":0.38},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.39700000000000002,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.376,"15min":0.38400000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.41099999999999998,"5min":0.50700000000000001,"15min":0.50700000000000001},"last":0.29199999999999998}]},{"osd":1,"last update":"Mon Mar 11 22:59:27 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.38900000000000001,"15min":0.39700000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.47599999999999998,"5min":0.53100000000000003,"15min":0.53100000000000003},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.39600000000000002,"15min":0.40400000000000003},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.46100000000000002,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.307}]},{"osd":3,"last update":"Mon Mar 11 22:58:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.59799999999999998}]},{"osd":4,"last update":"Mon Mar 11 22:59:18 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.55800000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 11 22:59:01 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.32800000000000001,"15min":0.32300000000000001},"min":{"1min":0.26200000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.35199999999999998,"15min":0.34399999999999997},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.25700000000000001}]},{"osd":2,"last update":"Mon Mar 11 22:59:28 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.39300000000000002,"15min":0.39100000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.308},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.38400000000000001,"15min":0.38200000000000001},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.23300000000000001}]},{"osd":3,"last update":"Mon Mar 11 22:58:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.622},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.55700000000000005}]},{"osd":4,"last update":"Mon Mar 11 22:59:17 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.57199999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]}]},{"osd":1,"up_from":14,"seq":60129542179,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 11 22:59:02 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.33000000000000002,"15min":0.32600000000000001},"min":{"1min":0.251,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.35699999999999998,"15min":0.35899999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.495,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.30199999999999999}]},{"osd":2,"last update":"Mon Mar 11 22:59:30 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.36599999999999999,"15min":0.37},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.504,"5min":0.51700000000000002,"15min":0.51700000000000002},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.38100000000000001,"15min":0.38400000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.55700000000000005,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.31900000000000001}]},{"osd":3,"last update":"Mon Mar 11 22:58:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.63},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.61699999999999999}]},{"osd":4,"last update":"Mon Mar 11 22:59:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.52600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:59:36.099 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-11T22:59:36.099 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-11T22:59:36.099 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-11T22:59:36.099 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph health --format=json 2024-03-11T22:59:36.364 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:59:36.725 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:36 smithi153 ceph-mon[28962]: from='client.14408 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:59:36.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:36 smithi163 ceph-mon[35584]: from='client.14408 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:59:37.660 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:37 smithi153 ceph-mon[28962]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:37.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:37 smithi163 ceph-mon[35584]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:37.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:37.975+0000 7f352253c700 1 -- 172.21.15.153:0/2827027358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f351c102ed0 msgr2=0x7f351c1032a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:37.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:37.975+0000 7f352253c700 1 --2- 172.21.15.153:0/2827027358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f351c102ed0 0x7f351c1032a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f350c004940 tx=0x7f350c009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:37.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:37.975+0000 7f352253c700 1 -- 172.21.15.153:0/2827027358 shutdown_connections 2024-03-11T22:59:37.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:37.975+0000 7f352253c700 1 --2- 172.21.15.153:0/2827027358 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f351c103870 0x7f351c0fcee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:37.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:37.975+0000 7f352253c700 1 --2- 172.21.15.153:0/2827027358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f351c102ed0 0x7f351c1032a0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:37.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:37.975+0000 7f352253c700 1 -- 172.21.15.153:0/2827027358 >> 172.21.15.153:0/2827027358 conn(0x7f351c0f8a40 msgr2=0x7f351c0fae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:37.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:37.975+0000 7f352253c700 1 -- 172.21.15.153:0/2827027358 shutdown_connections 2024-03-11T22:59:37.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:37.975+0000 7f352253c700 1 -- 172.21.15.153:0/2827027358 wait complete. 2024-03-11T22:59:37.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:37.976+0000 7f352253c700 1 Processor -- start 2024-03-11T22:59:37.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:37.976+0000 7f352253c700 1 -- start start 2024-03-11T22:59:37.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:37.976+0000 7f352253c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f351c102ed0 0x7f351c070290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:37.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:37.976+0000 7f352253c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f351c103870 0x7f351c0707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:37.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:37.976+0000 7f352253c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351c070dd0 con 0x7f351c102ed0 2024-03-11T22:59:37.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:37.976+0000 7f352253c700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351c070f40 con 0x7f351c103870 2024-03-11T22:59:37.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:37.976+0000 7f351bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f351c102ed0 0x7f351c070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:37.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:37.976+0000 7f351bfff700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f351c102ed0 0x7f351c070290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50022/0 (socket says 172.21.15.153:50022) 2024-03-11T22:59:37.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:37.976+0000 7f351bfff700 1 -- 172.21.15.153:0/921157916 learned_addr learned my addr 172.21.15.153:0/921157916 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:59:37.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:37.977+0000 7f351b7fe700 1 --2- 172.21.15.153:0/921157916 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f351c103870 0x7f351c0707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:37.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:37.977+0000 7f351bfff700 1 -- 172.21.15.153:0/921157916 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f351c103870 msgr2=0x7f351c0707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:37.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:37.977+0000 7f351bfff700 1 --2- 172.21.15.153:0/921157916 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f351c103870 0x7f351c0707d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:37.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:37.977+0000 7f351bfff700 1 -- 172.21.15.153:0/921157916 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350c005040 con 0x7f351c102ed0 2024-03-11T22:59:37.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:37.977+0000 7f351b7fe700 1 --2- 172.21.15.153:0/921157916 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f351c103870 0x7f351c0707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T22:59:37.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:37.977+0000 7f351bfff700 1 --2- 172.21.15.153:0/921157916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f351c102ed0 0x7f351c070290 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f350c009fd0 tx=0x7f350c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:37.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:37.977+0000 7f3518ff9700 1 -- 172.21.15.153:0/921157916 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f350c00e040 con 0x7f351c102ed0 2024-03-11T22:59:37.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:37.977+0000 7f3518ff9700 1 -- 172.21.15.153:0/921157916 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f350c00bbf0 con 0x7f351c102ed0 2024-03-11T22:59:37.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:37.977+0000 7f352253c700 1 -- 172.21.15.153:0/921157916 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f351c119bf0 con 0x7f351c102ed0 2024-03-11T22:59:37.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:37.977+0000 7f3518ff9700 1 -- 172.21.15.153:0/921157916 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f350c01c3f0 con 0x7f351c102ed0 2024-03-11T22:59:37.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:37.977+0000 7f352253c700 1 -- 172.21.15.153:0/921157916 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f351c11a0d0 con 0x7f351c102ed0 2024-03-11T22:59:37.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:37.978+0000 7f352253c700 1 -- 172.21.15.153:0/921157916 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f351c0fc660 con 0x7f351c102ed0 2024-03-11T22:59:37.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:37.979+0000 7f3518ff9700 1 -- 172.21.15.153:0/921157916 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f350c00bd60 con 0x7f351c102ed0 2024-03-11T22:59:37.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:37.979+0000 7f3518ff9700 1 --2- 172.21.15.153:0/921157916 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f35040692b0 0x7f350406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:37.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:37.979+0000 7f3518ff9700 1 -- 172.21.15.153:0/921157916 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f350c088160 con 0x7f351c102ed0 2024-03-11T22:59:37.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:37.979+0000 7f351b7fe700 1 --2- 172.21.15.153:0/921157916 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f35040692b0 0x7f350406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:37.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:37.979+0000 7f351b7fe700 1 --2- 172.21.15.153:0/921157916 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f35040692b0 0x7f350406b760 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3510009fd0 tx=0x7f351000b040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:37.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:37.983+0000 7f3518ff9700 1 -- 172.21.15.153:0/921157916 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f350c0547d0 con 0x7f351c102ed0 2024-03-11T22:59:38.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:38.206+0000 7f352253c700 1 -- 172.21.15.153:0/921157916 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f351c109fd0 con 0x7f351c102ed0 2024-03-11T22:59:38.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:38.206+0000 7f3518ff9700 1 -- 172.21.15.153:0/921157916 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f350c057df0 con 0x7f351c102ed0 2024-03-11T22:59:38.210 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T22:59:38.211 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-11T22:59:38.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:38.208+0000 7f352253c700 1 -- 172.21.15.153:0/921157916 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f35040692b0 msgr2=0x7f350406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:38.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:38.208+0000 7f352253c700 1 --2- 172.21.15.153:0/921157916 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f35040692b0 0x7f350406b760 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3510009fd0 tx=0x7f351000b040 comp rx=0 tx=0).stop 2024-03-11T22:59:38.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:38.208+0000 7f352253c700 1 -- 172.21.15.153:0/921157916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f351c102ed0 msgr2=0x7f351c070290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:38.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:38.208+0000 7f352253c700 1 --2- 172.21.15.153:0/921157916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f351c102ed0 0x7f351c070290 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f350c009fd0 tx=0x7f350c009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:38.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:38.208+0000 7f352253c700 1 -- 172.21.15.153:0/921157916 shutdown_connections 2024-03-11T22:59:38.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:38.208+0000 7f352253c700 1 --2- 172.21.15.153:0/921157916 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f351c102ed0 0x7f351c070290 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:38.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:38.208+0000 7f352253c700 1 --2- 172.21.15.153:0/921157916 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f35040692b0 0x7f350406b760 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:38.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:38.208+0000 7f352253c700 1 --2- 172.21.15.153:0/921157916 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f351c103870 0x7f351c0707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:38.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:38.208+0000 7f352253c700 1 -- 172.21.15.153:0/921157916 >> 172.21.15.153:0/921157916 conn(0x7f351c0f8a40 msgr2=0x7f351c0fa400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:38.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:38.208+0000 7f352253c700 1 -- 172.21.15.153:0/921157916 shutdown_connections 2024-03-11T22:59:38.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:38.208+0000 7f352253c700 1 -- 172.21.15.153:0/921157916 wait complete. 2024-03-11T22:59:38.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:38 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/921157916' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T22:59:38.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:38 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/921157916' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T22:59:38.896 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-11T22:59:38.896 INFO:tasks.cephadm:Setup complete, yielding 2024-03-11T22:59:38.897 INFO:teuthology.run_tasks:Running task print... 2024-03-11T22:59:38.907 INFO:teuthology.task.print:**** done end installing quincy cephadm ... 2024-03-11T22:59:38.908 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-11T22:59:38.919 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi153.front.sepia.ceph.com 2024-03-11T22:59:38.920 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-03-11T22:59:39.176 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:59:39.639 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:39 smithi153 ceph-mon[28962]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:39.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:39 smithi163 ceph-mon[35584]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:40.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:40.911+0000 7fbe74dd3700 1 -- 172.21.15.153:0/1729145349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe70102430 msgr2=0x7fbe70104810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:40.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:40.911+0000 7fbe74dd3700 1 --2- 172.21.15.153:0/1729145349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe70102430 0x7fbe70104810 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fbe58004940 tx=0x7fbe58009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:40.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:40.911+0000 7fbe74dd3700 1 -- 172.21.15.153:0/1729145349 shutdown_connections 2024-03-11T22:59:40.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:40.911+0000 7fbe74dd3700 1 --2- 172.21.15.153:0/1729145349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe70102430 0x7fbe70104810 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:40.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:40.911+0000 7fbe74dd3700 1 --2- 172.21.15.153:0/1729145349 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe700663e0 0x7fbe70101ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:40.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:40.911+0000 7fbe74dd3700 1 -- 172.21.15.153:0/1729145349 >> 172.21.15.153:0/1729145349 conn(0x7fbe700f74e0 msgr2=0x7fbe700f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:40.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:40.911+0000 7fbe74dd3700 1 -- 172.21.15.153:0/1729145349 shutdown_connections 2024-03-11T22:59:40.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:40.911+0000 7fbe74dd3700 1 -- 172.21.15.153:0/1729145349 wait complete. 2024-03-11T22:59:40.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:40.911+0000 7fbe74dd3700 1 Processor -- start 2024-03-11T22:59:40.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:40.911+0000 7fbe74dd3700 1 -- start start 2024-03-11T22:59:40.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:40.912+0000 7fbe74dd3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe700663e0 0x7fbe7010fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:40.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:40.912+0000 7fbe74dd3700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe70102430 0x7fbe701100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:40.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:40.912+0000 7fbe74dd3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe701106b0 con 0x7fbe700663e0 2024-03-11T22:59:40.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:40.912+0000 7fbe74dd3700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe70110f50 con 0x7fbe70102430 2024-03-11T22:59:40.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:40.912+0000 7fbe6e59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe700663e0 0x7fbe7010fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:40.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:40.912+0000 7fbe6e59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe700663e0 0x7fbe7010fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50054/0 (socket says 172.21.15.153:50054) 2024-03-11T22:59:40.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:40.912+0000 7fbe6e59c700 1 -- 172.21.15.153:0/41996713 learned_addr learned my addr 172.21.15.153:0/41996713 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:59:40.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:40.912+0000 7fbe6e59c700 1 -- 172.21.15.153:0/41996713 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe70102430 msgr2=0x7fbe701100b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:40.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:40.912+0000 7fbe6e59c700 1 --2- 172.21.15.153:0/41996713 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe70102430 0x7fbe701100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:40.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:40.912+0000 7fbe6e59c700 1 -- 172.21.15.153:0/41996713 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe58005040 con 0x7fbe700663e0 2024-03-11T22:59:40.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:40.912+0000 7fbe6e59c700 1 --2- 172.21.15.153:0/41996713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe700663e0 0x7fbe7010fb70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbe6000cb10 tx=0x7fbe60007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:40.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:40.912+0000 7fbe66ffd700 1 -- 172.21.15.153:0/41996713 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe60004200 con 0x7fbe700663e0 2024-03-11T22:59:40.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:40.912+0000 7fbe74dd3700 1 -- 172.21.15.153:0/41996713 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe70111230 con 0x7fbe700663e0 2024-03-11T22:59:40.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:40.912+0000 7fbe66ffd700 1 -- 172.21.15.153:0/41996713 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbe6000acc0 con 0x7fbe700663e0 2024-03-11T22:59:40.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:40.912+0000 7fbe66ffd700 1 -- 172.21.15.153:0/41996713 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe60003d90 con 0x7fbe700663e0 2024-03-11T22:59:40.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:40.912+0000 7fbe74dd3700 1 -- 172.21.15.153:0/41996713 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe70111770 con 0x7fbe700663e0 2024-03-11T22:59:40.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:40.913+0000 7fbe74dd3700 1 -- 172.21.15.153:0/41996713 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe7004bed0 con 0x7fbe700663e0 2024-03-11T22:59:40.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:40.914+0000 7fbe66ffd700 1 -- 172.21.15.153:0/41996713 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fbe60003430 con 0x7fbe700663e0 2024-03-11T22:59:40.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:40.914+0000 7fbe66ffd700 1 --2- 172.21.15.153:0/41996713 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fbe5c0692a0 0x7fbe5c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:40.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:40.914+0000 7fbe6dd9b700 1 --2- 172.21.15.153:0/41996713 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fbe5c0692a0 0x7fbe5c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:40.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:40.914+0000 7fbe66ffd700 1 -- 172.21.15.153:0/41996713 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fbe60087450 con 0x7fbe700663e0 2024-03-11T22:59:40.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:40.915+0000 7fbe6dd9b700 1 --2- 172.21.15.153:0/41996713 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fbe5c0692a0 0x7fbe5c06b750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbe58004f90 tx=0x7fbe58009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:40.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:40.919+0000 7fbe66ffd700 1 -- 172.21.15.153:0/41996713 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7fbe600574d0 con 0x7fbe700663e0 2024-03-11T22:59:41.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:41.102+0000 7fbe74dd3700 1 -- 172.21.15.153:0/41996713 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7fbe70107c10 con 0x7fbe700663e0 2024-03-11T22:59:41.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:41.106+0000 7fbe66ffd700 1 -- 172.21.15.153:0/41996713 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7fbe60057060 con 0x7fbe700663e0 2024-03-11T22:59:41.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:41.109+0000 7fbe74dd3700 1 -- 172.21.15.153:0/41996713 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fbe5c0692a0 msgr2=0x7fbe5c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:41.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:41.109+0000 7fbe74dd3700 1 --2- 172.21.15.153:0/41996713 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fbe5c0692a0 0x7fbe5c06b750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbe58004f90 tx=0x7fbe58009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:41.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:41.109+0000 7fbe74dd3700 1 -- 172.21.15.153:0/41996713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe700663e0 msgr2=0x7fbe7010fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:41.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:41.109+0000 7fbe74dd3700 1 --2- 172.21.15.153:0/41996713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe700663e0 0x7fbe7010fb70 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fbe6000cb10 tx=0x7fbe60007f90 comp rx=0 tx=0).stop 2024-03-11T22:59:41.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:41.109+0000 7fbe74dd3700 1 -- 172.21.15.153:0/41996713 shutdown_connections 2024-03-11T22:59:41.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:41.109+0000 7fbe74dd3700 1 --2- 172.21.15.153:0/41996713 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe700663e0 0x7fbe7010fb70 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:41.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:41.109+0000 7fbe74dd3700 1 --2- 172.21.15.153:0/41996713 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fbe5c0692a0 0x7fbe5c06b750 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:41.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:41.109+0000 7fbe74dd3700 1 --2- 172.21.15.153:0/41996713 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe70102430 0x7fbe701100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:41.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:41.109+0000 7fbe74dd3700 1 -- 172.21.15.153:0/41996713 >> 172.21.15.153:0/41996713 conn(0x7fbe700f74e0 msgr2=0x7fbe700f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:41.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:41.109+0000 7fbe74dd3700 1 -- 172.21.15.153:0/41996713 shutdown_connections 2024-03-11T22:59:41.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:41.109+0000 7fbe74dd3700 1 -- 172.21.15.153:0/41996713 wait complete. 2024-03-11T22:59:41.726 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:41 smithi153 ceph-mon[28962]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:41.726 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:41 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/41996713' entity='client.admin' 2024-03-11T22:59:41.726 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:41 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:59:41.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:41 smithi163 ceph-mon[35584]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:41.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:41 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/41996713' entity='client.admin' 2024-03-11T22:59:41.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:41 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:59:41.750 INFO:teuthology.run_tasks:Running task print... 2024-03-11T22:59:41.759 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-03-11T22:59:41.759 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-11T22:59:41.772 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi153.front.sepia.ceph.com 2024-03-11T22:59:41.772 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- bash -c 'ceph orch status' 2024-03-11T22:59:42.038 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:59:42.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:42 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:59:42.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:42 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:59:42.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:42 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:59:42.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:42 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:59:42.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:42 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:59:42.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:42 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T22:59:43.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:43 smithi153 ceph-mon[28962]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:43.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:43 smithi163 ceph-mon[35584]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:43.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.775+0000 7f04779c7700 1 -- 172.21.15.153:0/2705427348 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04700fcfb0 msgr2=0x7f04700fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:43.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.775+0000 7f04779c7700 1 --2- 172.21.15.153:0/2705427348 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04700fcfb0 0x7f04700fd420 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0460004940 tx=0x7f0460009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:43.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.775+0000 7f04779c7700 1 -- 172.21.15.153:0/2705427348 shutdown_connections 2024-03-11T22:59:43.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.775+0000 7f04779c7700 1 --2- 172.21.15.153:0/2705427348 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04700fcfb0 0x7f04700fd420 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:43.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.775+0000 7f04779c7700 1 --2- 172.21.15.153:0/2705427348 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04700fbe40 0x7f04700fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:43.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.775+0000 7f04779c7700 1 -- 172.21.15.153:0/2705427348 >> 172.21.15.153:0/2705427348 conn(0x7f04700f7500 msgr2=0x7f04700f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:43.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.775+0000 7f04779c7700 1 -- 172.21.15.153:0/2705427348 shutdown_connections 2024-03-11T22:59:43.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.775+0000 7f04779c7700 1 -- 172.21.15.153:0/2705427348 wait complete. 2024-03-11T22:59:43.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.775+0000 7f04779c7700 1 Processor -- start 2024-03-11T22:59:43.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.776+0000 7f04779c7700 1 -- start start 2024-03-11T22:59:43.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.776+0000 7f04779c7700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04700fbe40 0x7f047010d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:43.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.776+0000 7f04779c7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04700fcfb0 0x7f047010de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:43.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.776+0000 7f04779c7700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f047010e4f0 con 0x7f04700fcfb0 2024-03-11T22:59:43.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.776+0000 7f04779c7700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f047010ed30 con 0x7f04700fbe40 2024-03-11T22:59:43.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.776+0000 7f0474f62700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04700fcfb0 0x7f047010de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:43.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.776+0000 7f0474f62700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04700fcfb0 0x7f047010de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41808/0 (socket says 172.21.15.153:41808) 2024-03-11T22:59:43.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.776+0000 7f0474f62700 1 -- 172.21.15.153:0/3541492034 learned_addr learned my addr 172.21.15.153:0/3541492034 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:59:43.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.776+0000 7f0475763700 1 --2- 172.21.15.153:0/3541492034 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04700fbe40 0x7f047010d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:43.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.776+0000 7f0474f62700 1 -- 172.21.15.153:0/3541492034 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04700fbe40 msgr2=0x7f047010d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:43.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.776+0000 7f0474f62700 1 --2- 172.21.15.153:0/3541492034 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04700fbe40 0x7f047010d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:43.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.776+0000 7f0474f62700 1 -- 172.21.15.153:0/3541492034 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0460005040 con 0x7f04700fcfb0 2024-03-11T22:59:43.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.776+0000 7f0474f62700 1 --2- 172.21.15.153:0/3541492034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04700fcfb0 0x7f047010de60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f04600093b0 tx=0x7f0460009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:43.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.776+0000 7f0465ffb700 1 -- 172.21.15.153:0/3541492034 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f046000e040 con 0x7f04700fcfb0 2024-03-11T22:59:43.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.777+0000 7f04779c7700 1 -- 172.21.15.153:0/3541492034 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f047010efb0 con 0x7f04700fcfb0 2024-03-11T22:59:43.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.776+0000 7f0465ffb700 1 -- 172.21.15.153:0/3541492034 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f046000bcc0 con 0x7f04700fcfb0 2024-03-11T22:59:43.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.777+0000 7f0465ffb700 1 -- 172.21.15.153:0/3541492034 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f046001c630 con 0x7f04700fcfb0 2024-03-11T22:59:43.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.777+0000 7f04779c7700 1 -- 172.21.15.153:0/3541492034 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f047010f4f0 con 0x7f04700fcfb0 2024-03-11T22:59:43.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.778+0000 7f04779c7700 1 -- 172.21.15.153:0/3541492034 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f047004bed0 con 0x7f04700fcfb0 2024-03-11T22:59:43.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.778+0000 7f0465ffb700 1 -- 172.21.15.153:0/3541492034 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f04600139f0 con 0x7f04700fcfb0 2024-03-11T22:59:43.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.778+0000 7f0465ffb700 1 --2- 172.21.15.153:0/3541492034 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f045c0692a0 0x7f045c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:43.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.779+0000 7f0465ffb700 1 -- 172.21.15.153:0/3541492034 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0460088250 con 0x7f04700fcfb0 2024-03-11T22:59:43.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.779+0000 7f0475763700 1 --2- 172.21.15.153:0/3541492034 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f045c0692a0 0x7f045c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:43.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.779+0000 7f0475763700 1 --2- 172.21.15.153:0/3541492034 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f045c0692a0 0x7f045c06b750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0468001250 tx=0x7f0468009620 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:43.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.783+0000 7f0465ffb700 1 -- 172.21.15.153:0/3541492034 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f0460058350 con 0x7f04700fcfb0 2024-03-11T22:59:43.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.978+0000 7f04779c7700 1 -- 172.21.15.153:0/3541492034 --> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f047010f9f0 con 0x7f045c0692a0 2024-03-11T22:59:43.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.980+0000 7f0465ffb700 1 -- 172.21.15.153:0/3541492034 <== mgr.14204 v2:172.21.15.153:6800/29938766 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f047010f9f0 con 0x7f045c0692a0 2024-03-11T22:59:43.984 INFO:teuthology.orchestra.run.smithi153.stdout:Backend: cephadm 2024-03-11T22:59:43.984 INFO:teuthology.orchestra.run.smithi153.stdout:Available: Yes 2024-03-11T22:59:43.984 INFO:teuthology.orchestra.run.smithi153.stdout:Paused: No 2024-03-11T22:59:43.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.982+0000 7f04779c7700 1 -- 172.21.15.153:0/3541492034 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f045c0692a0 msgr2=0x7f045c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:43.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.982+0000 7f04779c7700 1 --2- 172.21.15.153:0/3541492034 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f045c0692a0 0x7f045c06b750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0468001250 tx=0x7f0468009620 comp rx=0 tx=0).stop 2024-03-11T22:59:43.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.982+0000 7f04779c7700 1 -- 172.21.15.153:0/3541492034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04700fcfb0 msgr2=0x7f047010de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:43.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.982+0000 7f04779c7700 1 --2- 172.21.15.153:0/3541492034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04700fcfb0 0x7f047010de60 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f04600093b0 tx=0x7f0460009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:43.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.982+0000 7f04779c7700 1 -- 172.21.15.153:0/3541492034 shutdown_connections 2024-03-11T22:59:43.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.982+0000 7f04779c7700 1 --2- 172.21.15.153:0/3541492034 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f045c0692a0 0x7f045c06b750 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:43.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.982+0000 7f04779c7700 1 --2- 172.21.15.153:0/3541492034 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f04700fbe40 0x7f047010d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:43.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.982+0000 7f04779c7700 1 --2- 172.21.15.153:0/3541492034 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f04700fcfb0 0x7f047010de60 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:43.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.982+0000 7f04779c7700 1 -- 172.21.15.153:0/3541492034 >> 172.21.15.153:0/3541492034 conn(0x7f04700f7500 msgr2=0x7f0470065f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:43.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.982+0000 7f04779c7700 1 -- 172.21.15.153:0/3541492034 shutdown_connections 2024-03-11T22:59:43.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:43.982+0000 7f04779c7700 1 -- 172.21.15.153:0/3541492034 wait complete. 2024-03-11T22:59:44.854 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-11T22:59:45.117 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:59:45.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:45 smithi153 ceph-mon[28962]: from='client.14420 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:59:45.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:45 smithi153 ceph-mon[28962]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:45.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:45 smithi163 ceph-mon[35584]: from='client.14420 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:59:45.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:45 smithi163 ceph-mon[35584]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:46.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.619+0000 7fbb402a2700 1 -- 172.21.15.153:0/3671494474 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb380fcf30 msgr2=0x7fbb380fd340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:46.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.619+0000 7fbb402a2700 1 --2- 172.21.15.153:0/3671494474 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb380fcf30 0x7fbb380fd340 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbb34004940 tx=0x7fbb34009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:46.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.619+0000 7fbb402a2700 1 -- 172.21.15.153:0/3671494474 shutdown_connections 2024-03-11T22:59:46.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.619+0000 7fbb402a2700 1 --2- 172.21.15.153:0/3671494474 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb380fe1e0 0x7fbb380fe650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:46.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.619+0000 7fbb402a2700 1 --2- 172.21.15.153:0/3671494474 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb380fcf30 0x7fbb380fd340 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:46.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.619+0000 7fbb402a2700 1 -- 172.21.15.153:0/3671494474 >> 172.21.15.153:0/3671494474 conn(0x7fbb380f86b0 msgr2=0x7fbb380faac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:46.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.619+0000 7fbb402a2700 1 -- 172.21.15.153:0/3671494474 shutdown_connections 2024-03-11T22:59:46.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.619+0000 7fbb402a2700 1 -- 172.21.15.153:0/3671494474 wait complete. 2024-03-11T22:59:46.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.620+0000 7fbb402a2700 1 Processor -- start 2024-03-11T22:59:46.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.620+0000 7fbb402a2700 1 -- start start 2024-03-11T22:59:46.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.620+0000 7fbb402a2700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb380fcf30 0x7fbb3810fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:46.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.620+0000 7fbb402a2700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb380fe1e0 0x7fbb381100a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:46.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.620+0000 7fbb402a2700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb38110730 con 0x7fbb380fe1e0 2024-03-11T22:59:46.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.620+0000 7fbb402a2700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb38110f70 con 0x7fbb380fcf30 2024-03-11T22:59:46.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.620+0000 7fbb3d83d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb380fe1e0 0x7fbb381100a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:46.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.620+0000 7fbb3d83d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb380fe1e0 0x7fbb381100a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41830/0 (socket says 172.21.15.153:41830) 2024-03-11T22:59:46.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.620+0000 7fbb3d83d700 1 -- 172.21.15.153:0/876175104 learned_addr learned my addr 172.21.15.153:0/876175104 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:59:46.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.620+0000 7fbb3e03e700 1 --2- 172.21.15.153:0/876175104 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb380fcf30 0x7fbb3810fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:46.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.620+0000 7fbb3d83d700 1 -- 172.21.15.153:0/876175104 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb380fcf30 msgr2=0x7fbb3810fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:46.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.620+0000 7fbb3d83d700 1 --2- 172.21.15.153:0/876175104 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb380fcf30 0x7fbb3810fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:46.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.620+0000 7fbb3d83d700 1 -- 172.21.15.153:0/876175104 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb34005040 con 0x7fbb380fe1e0 2024-03-11T22:59:46.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.620+0000 7fbb3d83d700 1 --2- 172.21.15.153:0/876175104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb380fe1e0 0x7fbb381100a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fbb2800b790 tx=0x7fbb28009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:46.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.621+0000 7fbb2effd700 1 -- 172.21.15.153:0/876175104 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb2800f6f0 con 0x7fbb380fe1e0 2024-03-11T22:59:46.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.621+0000 7fbb2effd700 1 -- 172.21.15.153:0/876175104 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbb2800fd20 con 0x7fbb380fe1e0 2024-03-11T22:59:46.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.621+0000 7fbb402a2700 1 -- 172.21.15.153:0/876175104 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb38111250 con 0x7fbb380fe1e0 2024-03-11T22:59:46.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.621+0000 7fbb2effd700 1 -- 172.21.15.153:0/876175104 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb280103a0 con 0x7fbb380fe1e0 2024-03-11T22:59:46.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.621+0000 7fbb402a2700 1 -- 172.21.15.153:0/876175104 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb38111790 con 0x7fbb380fe1e0 2024-03-11T22:59:46.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.622+0000 7fbb402a2700 1 -- 172.21.15.153:0/876175104 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb38102280 con 0x7fbb380fe1e0 2024-03-11T22:59:46.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.622+0000 7fbb2effd700 1 -- 172.21.15.153:0/876175104 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fbb28011400 con 0x7fbb380fe1e0 2024-03-11T22:59:46.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.622+0000 7fbb2effd700 1 --2- 172.21.15.153:0/876175104 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fbb240692b0 0x7fbb2406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:46.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.622+0000 7fbb2effd700 1 -- 172.21.15.153:0/876175104 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fbb28086b50 con 0x7fbb380fe1e0 2024-03-11T22:59:46.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.622+0000 7fbb3e03e700 1 --2- 172.21.15.153:0/876175104 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fbb240692b0 0x7fbb2406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:46.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.623+0000 7fbb3e03e700 1 --2- 172.21.15.153:0/876175104 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fbb240692b0 0x7fbb2406b760 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbb3400b7a0 tx=0x7fbb34009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:46.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.627+0000 7fbb2effd700 1 -- 172.21.15.153:0/876175104 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7fbb28056cf0 con 0x7fbb380fe1e0 2024-03-11T22:59:46.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.825+0000 7fbb402a2700 1 -- 172.21.15.153:0/876175104 --> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb38111ac0 con 0x7fbb240692b0 2024-03-11T22:59:46.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.833+0000 7fbb2effd700 1 -- 172.21.15.153:0/876175104 <== mgr.14204 v2:172.21.15.153:6800/29938766 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2682 (secure 0 0 0) 0x7fbb38111ac0 con 0x7fbb240692b0 2024-03-11T22:59:46.838 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T22:59:46.838 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (3m) 2m ago 5m 26.5M - 0.25.0 c8568f914cd2 7b54ac6814f8 2024-03-11T22:59:46.838 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (5m) 2m ago 5m 6665k - 17.2.7-620-gc186d0ca a92afbb1c283 9a75910c50de 2024-03-11T22:59:46.838 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi163 smithi163 running (4m) 68s ago 4m 6660k - 17.2.7-620-gc186d0ca a92afbb1c283 8f8506abaa8f 2024-03-11T22:59:46.838 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (3m) 2m ago 4m 82.5M - 9.4.7 954c08fa6188 3fd24b741053 2024-03-11T22:59:46.838 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.nyocep smithi153 *:9283 running (6m) 2m ago 6m 472M - 17.2.7-620-gc186d0ca a92afbb1c283 fb9a27cf063c 2024-03-11T22:59:46.838 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi163.ivtqrh smithi163 *:8443,9283 running (4m) 68s ago 4m 436M - 17.2.7-620-gc186d0ca a92afbb1c283 8584d0f7d4fa 2024-03-11T22:59:46.838 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (6m) 2m ago 6m 42.1M 2048M 17.2.7-620-gc186d0ca a92afbb1c283 0dad3ee1fdff 2024-03-11T22:59:46.838 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi163 smithi163 running (4m) 68s ago 4m 38.1M 2048M 17.2.7-620-gc186d0ca a92afbb1c283 fadb9d7bcbe8 2024-03-11T22:59:46.838 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (5m) 2m ago 5m 16.1M - 1.5.0 0da6a335fe13 cddc4be66b19 2024-03-11T22:59:46.839 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi163 smithi163 *:9100 running (4m) 68s ago 4m 17.0M - 1.5.0 0da6a335fe13 0aa9766c249d 2024-03-11T22:59:46.839 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (3m) 2m ago 3m 39.5M 4288M 17.2.7-620-gc186d0ca a92afbb1c283 e1d999ae7d94 2024-03-11T22:59:46.839 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (2m) 2m ago 2m 39.4M 4288M 17.2.7-620-gc186d0ca a92afbb1c283 23e1e01405ed 2024-03-11T22:59:46.839 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (2m) 2m ago 2m 37.4M 4288M 17.2.7-620-gc186d0ca a92afbb1c283 74c3d447a069 2024-03-11T22:59:46.839 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi163 running (2m) 68s ago 2m 39.2M 5312M 17.2.7-620-gc186d0ca a92afbb1c283 2b06c7febdd0 2024-03-11T22:59:46.839 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi163 running (99s) 68s ago 98s 36.4M 5312M 17.2.7-620-gc186d0ca a92afbb1c283 6e1f1b2171fb 2024-03-11T22:59:46.839 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi163 running (73s) 68s ago 73s 10.0M 5312M 17.2.7-620-gc186d0ca a92afbb1c283 c5d707e483c5 2024-03-11T22:59:46.839 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi153 smithi153 *:9095 running (3m) 2m ago 4m 35.7M - 2.43.0 a07b618ecd1d 7a3323c1a837 2024-03-11T22:59:46.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.835+0000 7fbb402a2700 1 -- 172.21.15.153:0/876175104 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fbb240692b0 msgr2=0x7fbb2406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:46.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.835+0000 7fbb402a2700 1 --2- 172.21.15.153:0/876175104 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fbb240692b0 0x7fbb2406b760 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbb3400b7a0 tx=0x7fbb34009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:46.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.835+0000 7fbb402a2700 1 -- 172.21.15.153:0/876175104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb380fe1e0 msgr2=0x7fbb381100a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:46.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.835+0000 7fbb402a2700 1 --2- 172.21.15.153:0/876175104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb380fe1e0 0x7fbb381100a0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fbb2800b790 tx=0x7fbb28009250 comp rx=0 tx=0).stop 2024-03-11T22:59:46.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.835+0000 7fbb402a2700 1 -- 172.21.15.153:0/876175104 shutdown_connections 2024-03-11T22:59:46.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.835+0000 7fbb402a2700 1 --2- 172.21.15.153:0/876175104 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fbb240692b0 0x7fbb2406b760 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:46.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.835+0000 7fbb402a2700 1 --2- 172.21.15.153:0/876175104 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb380fcf30 0x7fbb3810fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:46.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.835+0000 7fbb402a2700 1 --2- 172.21.15.153:0/876175104 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbb380fe1e0 0x7fbb381100a0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:46.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.835+0000 7fbb402a2700 1 -- 172.21.15.153:0/876175104 >> 172.21.15.153:0/876175104 conn(0x7fbb380f86b0 msgr2=0x7fbb380ff400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:46.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.835+0000 7fbb402a2700 1 -- 172.21.15.153:0/876175104 shutdown_connections 2024-03-11T22:59:46.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:46.835+0000 7fbb402a2700 1 -- 172.21.15.153:0/876175104 wait complete. 2024-03-11T22:59:47.513 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:47 smithi153 ceph-mon[28962]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:47.514 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-11T22:59:47.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:47 smithi163 ceph-mon[35584]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:47.771 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:59:48.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:48 smithi163 ceph-mon[35584]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:59:48.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:48 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:59:48.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:48 smithi153 ceph-mon[28962]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:59:48.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:48 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:59:49.516 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:49 smithi153 ceph-mon[28962]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:49.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.512+0000 7f8f75d36700 1 -- 172.21.15.153:0/1841512735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f70071e90 msgr2=0x7f8f70070300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:49.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.512+0000 7f8f75d36700 1 --2- 172.21.15.153:0/1841512735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f70071e90 0x7f8f70070300 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8f68004940 tx=0x7f8f68009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:49.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.512+0000 7f8f75d36700 1 -- 172.21.15.153:0/1841512735 shutdown_connections 2024-03-11T22:59:49.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.512+0000 7f8f75d36700 1 --2- 172.21.15.153:0/1841512735 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f70070840 0x7f8f70070cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:49.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.512+0000 7f8f75d36700 1 --2- 172.21.15.153:0/1841512735 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f70071e90 0x7f8f70070300 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:49.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.512+0000 7f8f75d36700 1 -- 172.21.15.153:0/1841512735 >> 172.21.15.153:0/1841512735 conn(0x7f8f700f8620 msgr2=0x7f8f700faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:49.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.512+0000 7f8f75d36700 1 -- 172.21.15.153:0/1841512735 shutdown_connections 2024-03-11T22:59:49.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.512+0000 7f8f75d36700 1 -- 172.21.15.153:0/1841512735 wait complete. 2024-03-11T22:59:49.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.512+0000 7f8f75d36700 1 Processor -- start 2024-03-11T22:59:49.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.513+0000 7f8f75d36700 1 -- start start 2024-03-11T22:59:49.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.513+0000 7f8f75d36700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f70070840 0x7f8f7010fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:49.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.513+0000 7f8f75d36700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f70071e90 0x7f8f701100a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:49.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.513+0000 7f8f75d36700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f701106a0 con 0x7f8f70070840 2024-03-11T22:59:49.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.513+0000 7f8f75d36700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f70110f40 con 0x7f8f70071e90 2024-03-11T22:59:49.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.513+0000 7f8f6f7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f70070840 0x7f8f7010fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:49.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.513+0000 7f8f6f7fe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f70070840 0x7f8f7010fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41858/0 (socket says 172.21.15.153:41858) 2024-03-11T22:59:49.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.513+0000 7f8f6f7fe700 1 -- 172.21.15.153:0/357973899 learned_addr learned my addr 172.21.15.153:0/357973899 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:59:49.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.513+0000 7f8f6effd700 1 --2- 172.21.15.153:0/357973899 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f70071e90 0x7f8f701100a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:49.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.513+0000 7f8f6f7fe700 1 -- 172.21.15.153:0/357973899 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f70071e90 msgr2=0x7f8f701100a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:49.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.513+0000 7f8f6f7fe700 1 --2- 172.21.15.153:0/357973899 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f70071e90 0x7f8f701100a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:49.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.513+0000 7f8f6f7fe700 1 -- 172.21.15.153:0/357973899 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f68005040 con 0x7f8f70070840 2024-03-11T22:59:49.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.513+0000 7f8f6f7fe700 1 --2- 172.21.15.153:0/357973899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f70070840 0x7f8f7010fb60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8f680093b0 tx=0x7f8f68009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:49.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.514+0000 7f8f57fff700 1 -- 172.21.15.153:0/357973899 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f6800e040 con 0x7f8f70070840 2024-03-11T22:59:49.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.514+0000 7f8f75d36700 1 -- 172.21.15.153:0/357973899 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f701111c0 con 0x7f8f70070840 2024-03-11T22:59:49.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.514+0000 7f8f57fff700 1 -- 172.21.15.153:0/357973899 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8f6800bcc0 con 0x7f8f70070840 2024-03-11T22:59:49.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.514+0000 7f8f57fff700 1 -- 172.21.15.153:0/357973899 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f6801c630 con 0x7f8f70070840 2024-03-11T22:59:49.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.514+0000 7f8f75d36700 1 -- 172.21.15.153:0/357973899 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f70111700 con 0x7f8f70070840 2024-03-11T22:59:49.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.515+0000 7f8f75d36700 1 -- 172.21.15.153:0/357973899 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f7004bed0 con 0x7f8f70070840 2024-03-11T22:59:49.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.515+0000 7f8f57fff700 1 -- 172.21.15.153:0/357973899 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f8f6801c8a0 con 0x7f8f70070840 2024-03-11T22:59:49.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.515+0000 7f8f57fff700 1 --2- 172.21.15.153:0/357973899 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f8f580692a0 0x7f8f5806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:49.520 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.516+0000 7f8f57fff700 1 -- 172.21.15.153:0/357973899 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8f68088290 con 0x7f8f70070840 2024-03-11T22:59:49.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.516+0000 7f8f6effd700 1 --2- 172.21.15.153:0/357973899 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f8f580692a0 0x7f8f5806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:49.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.516+0000 7f8f6effd700 1 --2- 172.21.15.153:0/357973899 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f8f580692a0 0x7f8f5806b750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8f64004fa0 tx=0x7f8f6400b040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:49.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.520+0000 7f8f57fff700 1 -- 172.21.15.153:0/357973899 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f8f68058390 con 0x7f8f70070840 2024-03-11T22:59:49.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.718+0000 7f8f75d36700 1 -- 172.21.15.153:0/357973899 --> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f8f70111350 con 0x7f8f580692a0 2024-03-11T22:59:49.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.723+0000 7f8f57fff700 1 -- 172.21.15.153:0/357973899 <== mgr.14204 v2:172.21.15.153:6800/29938766 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1179 (secure 0 0 0) 0x7f8f70111350 con 0x7f8f580692a0 2024-03-11T22:59:49.727 INFO:teuthology.orchestra.run.smithi153.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-11T22:59:49.727 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager ?:9093,9094 1/1 2m ago 5m count:1 2024-03-11T22:59:49.727 INFO:teuthology.orchestra.run.smithi153.stdout:crash 2/2 2m ago 5m * 2024-03-11T22:59:49.727 INFO:teuthology.orchestra.run.smithi153.stdout:grafana ?:3000 1/1 2m ago 5m count:1 2024-03-11T22:59:49.727 INFO:teuthology.orchestra.run.smithi153.stdout:mgr 2/2 2m ago 5m count:2 2024-03-11T22:59:49.727 INFO:teuthology.orchestra.run.smithi153.stdout:mon 2/2 2m ago 4m smithi153:172.21.15.153=smithi153;smithi163:172.21.15.163=smithi163;count:2 2024-03-11T22:59:49.727 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter ?:9100 2/2 2m ago 5m * 2024-03-11T22:59:49.727 INFO:teuthology.orchestra.run.smithi153.stdout:osd 6 2m ago - 2024-03-11T22:59:49.727 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus ?:9095 1/1 2m ago 5m count:1 2024-03-11T22:59:49.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.725+0000 7f8f75d36700 1 -- 172.21.15.153:0/357973899 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f8f580692a0 msgr2=0x7f8f5806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:49.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.725+0000 7f8f75d36700 1 --2- 172.21.15.153:0/357973899 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f8f580692a0 0x7f8f5806b750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8f64004fa0 tx=0x7f8f6400b040 comp rx=0 tx=0).stop 2024-03-11T22:59:49.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.725+0000 7f8f75d36700 1 -- 172.21.15.153:0/357973899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f70070840 msgr2=0x7f8f7010fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:49.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.725+0000 7f8f75d36700 1 --2- 172.21.15.153:0/357973899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f70070840 0x7f8f7010fb60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8f680093b0 tx=0x7f8f68009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:49.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.725+0000 7f8f75d36700 1 -- 172.21.15.153:0/357973899 shutdown_connections 2024-03-11T22:59:49.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.725+0000 7f8f75d36700 1 --2- 172.21.15.153:0/357973899 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f70070840 0x7f8f7010fb60 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:49.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.725+0000 7f8f75d36700 1 --2- 172.21.15.153:0/357973899 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f8f580692a0 0x7f8f5806b750 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:49.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.725+0000 7f8f75d36700 1 --2- 172.21.15.153:0/357973899 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f70071e90 0x7f8f701100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:49.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.725+0000 7f8f75d36700 1 -- 172.21.15.153:0/357973899 >> 172.21.15.153:0/357973899 conn(0x7f8f700f8620 msgr2=0x7f8f70102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:49.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.725+0000 7f8f75d36700 1 -- 172.21.15.153:0/357973899 shutdown_connections 2024-03-11T22:59:49.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:49.725+0000 7f8f75d36700 1 -- 172.21.15.153:0/357973899 wait complete. 2024-03-11T22:59:49.737 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:49 smithi163 ceph-mon[35584]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:50.396 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-11T22:59:50.653 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:59:51.703 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:51 smithi153 ceph-mon[28962]: from='client.14428 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:59:51.704 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:51 smithi153 ceph-mon[28962]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:51.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:51 smithi163 ceph-mon[35584]: from='client.14428 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:59:51.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:51 smithi163 ceph-mon[35584]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:52.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.167+0000 7f710c353700 1 -- 172.21.15.153:0/2339118839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71040fcfb0 msgr2=0x7f71040fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:52.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.167+0000 7f710c353700 1 --2- 172.21.15.153:0/2339118839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71040fcfb0 0x7f71040fd420 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f70f4004940 tx=0x7f70f4009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:52.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.167+0000 7f710c353700 1 -- 172.21.15.153:0/2339118839 shutdown_connections 2024-03-11T22:59:52.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.167+0000 7f710c353700 1 --2- 172.21.15.153:0/2339118839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71040fcfb0 0x7f71040fd420 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:52.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.167+0000 7f710c353700 1 --2- 172.21.15.153:0/2339118839 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71040fbe40 0x7f71040fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:52.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.167+0000 7f710c353700 1 -- 172.21.15.153:0/2339118839 >> 172.21.15.153:0/2339118839 conn(0x7f71040f74e0 msgr2=0x7f71040f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:52.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.167+0000 7f710c353700 1 -- 172.21.15.153:0/2339118839 shutdown_connections 2024-03-11T22:59:52.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.167+0000 7f710c353700 1 -- 172.21.15.153:0/2339118839 wait complete. 2024-03-11T22:59:52.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.167+0000 7f710c353700 1 Processor -- start 2024-03-11T22:59:52.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.168+0000 7f710c353700 1 -- start start 2024-03-11T22:59:52.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.168+0000 7f710c353700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71040fbe40 0x7f7104112a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:52.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.168+0000 7f710c353700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71040fcfb0 0x7f7104114fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:52.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.168+0000 7f710c353700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7104066350 con 0x7f71040fbe40 2024-03-11T22:59:52.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.168+0000 7f710c353700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7104066490 con 0x7f71040fcfb0 2024-03-11T22:59:52.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.168+0000 7f710a0ef700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71040fbe40 0x7f7104112a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:52.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.168+0000 7f710a0ef700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71040fbe40 0x7f7104112a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53944/0 (socket says 172.21.15.153:53944) 2024-03-11T22:59:52.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.168+0000 7f710a0ef700 1 -- 172.21.15.153:0/3090258674 learned_addr learned my addr 172.21.15.153:0/3090258674 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:59:52.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.168+0000 7f71098ee700 1 --2- 172.21.15.153:0/3090258674 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71040fcfb0 0x7f7104114fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:52.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.168+0000 7f710a0ef700 1 -- 172.21.15.153:0/3090258674 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71040fcfb0 msgr2=0x7f7104114fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:52.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.168+0000 7f710a0ef700 1 --2- 172.21.15.153:0/3090258674 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71040fcfb0 0x7f7104114fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:52.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.168+0000 7f710a0ef700 1 -- 172.21.15.153:0/3090258674 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f4005040 con 0x7f71040fbe40 2024-03-11T22:59:52.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.168+0000 7f710a0ef700 1 --2- 172.21.15.153:0/3090258674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71040fbe40 0x7f7104112a90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f710000c830 tx=0x7f7100004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:52.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.168+0000 7f70faffd700 1 -- 172.21.15.153:0/3090258674 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f710000a760 con 0x7f71040fbe40 2024-03-11T22:59:52.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.168+0000 7f710c353700 1 -- 172.21.15.153:0/3090258674 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7104115520 con 0x7f71040fbe40 2024-03-11T22:59:52.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.168+0000 7f70faffd700 1 -- 172.21.15.153:0/3090258674 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f710000ad90 con 0x7f71040fbe40 2024-03-11T22:59:52.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.169+0000 7f70faffd700 1 -- 172.21.15.153:0/3090258674 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71000113a0 con 0x7f71040fbe40 2024-03-11T22:59:52.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.169+0000 7f710c353700 1 -- 172.21.15.153:0/3090258674 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7104115920 con 0x7f71040fbe40 2024-03-11T22:59:52.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.169+0000 7f710c353700 1 -- 172.21.15.153:0/3090258674 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f710404bed0 con 0x7f71040fbe40 2024-03-11T22:59:52.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.170+0000 7f70faffd700 1 -- 172.21.15.153:0/3090258674 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f7100012400 con 0x7f71040fbe40 2024-03-11T22:59:52.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.170+0000 7f70faffd700 1 --2- 172.21.15.153:0/3090258674 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f70f00692a0 0x7f70f006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:52.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.170+0000 7f71098ee700 1 --2- 172.21.15.153:0/3090258674 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f70f00692a0 0x7f70f006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:52.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.170+0000 7f70faffd700 1 -- 172.21.15.153:0/3090258674 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7100086ef0 con 0x7f71040fbe40 2024-03-11T22:59:52.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.171+0000 7f71098ee700 1 --2- 172.21.15.153:0/3090258674 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f70f00692a0 0x7f70f006b750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f70f4004f90 tx=0x7f70f4009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:52.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.175+0000 7f70faffd700 1 -- 172.21.15.153:0/3090258674 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f7100053560 con 0x7f71040fbe40 2024-03-11T22:59:52.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.366+0000 7f710c353700 1 -- 172.21.15.153:0/3090258674 --> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f7104101990 con 0x7f70f00692a0 2024-03-11T22:59:52.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.367+0000 7f70faffd700 1 -- 172.21.15.153:0/3090258674 <== mgr.14204 v2:172.21.15.153:6800/29938766 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7f7104101990 con 0x7f70f00692a0 2024-03-11T22:59:52.371 INFO:teuthology.orchestra.run.smithi153.stdout:HOST ADDR LABELS STATUS 2024-03-11T22:59:52.372 INFO:teuthology.orchestra.run.smithi153.stdout:smithi153 172.21.15.153 2024-03-11T22:59:52.372 INFO:teuthology.orchestra.run.smithi153.stdout:smithi163 172.21.15.163 2024-03-11T22:59:52.372 INFO:teuthology.orchestra.run.smithi153.stdout:2 hosts in cluster 2024-03-11T22:59:52.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.369+0000 7f710c353700 1 -- 172.21.15.153:0/3090258674 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f70f00692a0 msgr2=0x7f70f006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:52.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.369+0000 7f710c353700 1 --2- 172.21.15.153:0/3090258674 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f70f00692a0 0x7f70f006b750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f70f4004f90 tx=0x7f70f4009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:52.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.369+0000 7f710c353700 1 -- 172.21.15.153:0/3090258674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71040fbe40 msgr2=0x7f7104112a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:52.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.369+0000 7f710c353700 1 --2- 172.21.15.153:0/3090258674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71040fbe40 0x7f7104112a90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f710000c830 tx=0x7f7100004470 comp rx=0 tx=0).stop 2024-03-11T22:59:52.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.369+0000 7f710c353700 1 -- 172.21.15.153:0/3090258674 shutdown_connections 2024-03-11T22:59:52.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.369+0000 7f710c353700 1 --2- 172.21.15.153:0/3090258674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71040fbe40 0x7f7104112a90 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:52.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.369+0000 7f710c353700 1 --2- 172.21.15.153:0/3090258674 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f70f00692a0 0x7f70f006b750 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:52.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.369+0000 7f710c353700 1 --2- 172.21.15.153:0/3090258674 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f71040fcfb0 0x7f7104114fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:52.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.369+0000 7f710c353700 1 -- 172.21.15.153:0/3090258674 >> 172.21.15.153:0/3090258674 conn(0x7f71040f74e0 msgr2=0x7f7104100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:52.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.369+0000 7f710c353700 1 -- 172.21.15.153:0/3090258674 shutdown_connections 2024-03-11T22:59:52.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:52.369+0000 7f710c353700 1 -- 172.21.15.153:0/3090258674 wait complete. 2024-03-11T22:59:53.019 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-11T22:59:53.275 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:59:53.649 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:53 smithi153 ceph-mon[28962]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:53.650 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:53 smithi153 ceph-mon[28962]: from='client.14432 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:59:53.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:53 smithi163 ceph-mon[35584]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:53.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:53 smithi163 ceph-mon[35584]: from='client.14432 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:59:54.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:54.949+0000 7f8a7101c700 1 -- 172.21.15.153:0/1568486869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a6c0fbe40 msgr2=0x7f8a6c0fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:54.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:54.949+0000 7f8a7101c700 1 --2- 172.21.15.153:0/1568486869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a6c0fbe40 0x7f8a6c0fc250 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c004940 tx=0x7f8a5c009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:54.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:54.949+0000 7f8a7101c700 1 -- 172.21.15.153:0/1568486869 shutdown_connections 2024-03-11T22:59:54.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:54.949+0000 7f8a7101c700 1 --2- 172.21.15.153:0/1568486869 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a6c0fcfb0 0x7f8a6c0fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:54.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:54.949+0000 7f8a7101c700 1 --2- 172.21.15.153:0/1568486869 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a6c0fbe40 0x7f8a6c0fc250 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:54.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:54.949+0000 7f8a7101c700 1 -- 172.21.15.153:0/1568486869 >> 172.21.15.153:0/1568486869 conn(0x7f8a6c0f7520 msgr2=0x7f8a6c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:54.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:54.949+0000 7f8a7101c700 1 -- 172.21.15.153:0/1568486869 shutdown_connections 2024-03-11T22:59:54.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:54.949+0000 7f8a7101c700 1 -- 172.21.15.153:0/1568486869 wait complete. 2024-03-11T22:59:54.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:54.950+0000 7f8a7101c700 1 Processor -- start 2024-03-11T22:59:54.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:54.950+0000 7f8a7101c700 1 -- start start 2024-03-11T22:59:54.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:54.950+0000 7f8a7101c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a6c0fbe40 0x7f8a6c10c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:54.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:54.950+0000 7f8a7101c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a6c0fcfb0 0x7f8a6c10cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:54.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:54.950+0000 7f8a7101c700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6c10d370 con 0x7f8a6c0fbe40 2024-03-11T22:59:54.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:54.950+0000 7f8a7101c700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6c10dc10 con 0x7f8a6c0fcfb0 2024-03-11T22:59:54.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:54.950+0000 7f8a6ad9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a6c0fbe40 0x7f8a6c10c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:54.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:54.950+0000 7f8a6ad9d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a6c0fbe40 0x7f8a6c10c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53962/0 (socket says 172.21.15.153:53962) 2024-03-11T22:59:54.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:54.950+0000 7f8a6ad9d700 1 -- 172.21.15.153:0/925456712 learned_addr learned my addr 172.21.15.153:0/925456712 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:59:54.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:54.950+0000 7f8a6a59c700 1 --2- 172.21.15.153:0/925456712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a6c0fcfb0 0x7f8a6c10cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:54.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:54.950+0000 7f8a6ad9d700 1 -- 172.21.15.153:0/925456712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a6c0fcfb0 msgr2=0x7f8a6c10cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:54.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:54.950+0000 7f8a6ad9d700 1 --2- 172.21.15.153:0/925456712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a6c0fcfb0 0x7f8a6c10cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:54.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:54.950+0000 7f8a6ad9d700 1 -- 172.21.15.153:0/925456712 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a5c005040 con 0x7f8a6c0fbe40 2024-03-11T22:59:54.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:54.950+0000 7f8a6ad9d700 1 --2- 172.21.15.153:0/925456712 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a6c0fbe40 0x7f8a6c10c830 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c0093b0 tx=0x7f8a5c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:54.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:54.950+0000 7f8a537fe700 1 -- 172.21.15.153:0/925456712 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a5c00e040 con 0x7f8a6c0fbe40 2024-03-11T22:59:54.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:54.951+0000 7f8a537fe700 1 -- 172.21.15.153:0/925456712 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a5c00bcc0 con 0x7f8a6c0fbe40 2024-03-11T22:59:54.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:54.951+0000 7f8a7101c700 1 -- 172.21.15.153:0/925456712 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a6c10de90 con 0x7f8a6c0fbe40 2024-03-11T22:59:54.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:54.951+0000 7f8a537fe700 1 -- 172.21.15.153:0/925456712 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a5c01c630 con 0x7f8a6c0fbe40 2024-03-11T22:59:54.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:54.951+0000 7f8a7101c700 1 -- 172.21.15.153:0/925456712 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a6c10e3d0 con 0x7f8a6c0fbe40 2024-03-11T22:59:54.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:54.951+0000 7f8a7101c700 1 -- 172.21.15.153:0/925456712 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a6c04bed0 con 0x7f8a6c0fbe40 2024-03-11T22:59:54.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:54.952+0000 7f8a537fe700 1 -- 172.21.15.153:0/925456712 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f8a5c0139f0 con 0x7f8a6c0fbe40 2024-03-11T22:59:54.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:54.952+0000 7f8a537fe700 1 --2- 172.21.15.153:0/925456712 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f8a540692a0 0x7f8a5406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:54.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:54.952+0000 7f8a537fe700 1 -- 172.21.15.153:0/925456712 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8a5c088250 con 0x7f8a6c0fbe40 2024-03-11T22:59:54.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:54.952+0000 7f8a6a59c700 1 --2- 172.21.15.153:0/925456712 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f8a540692a0 0x7f8a5406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:54.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:54.953+0000 7f8a6a59c700 1 --2- 172.21.15.153:0/925456712 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f8a540692a0 0x7f8a5406b750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8a60007fd0 tx=0x7f8a60009040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:54.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:54.957+0000 7f8a537fe700 1 -- 172.21.15.153:0/925456712 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f8a5c058350 con 0x7f8a6c0fbe40 2024-03-11T22:59:55.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:55.151+0000 7f8a7101c700 1 -- 172.21.15.153:0/925456712 --> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f8a6c101990 con 0x7f8a540692a0 2024-03-11T22:59:55.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:55.153+0000 7f8a537fe700 1 -- 172.21.15.153:0/925456712 <== mgr.14204 v2:172.21.15.153:6800/29938766 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7f8a6c101990 con 0x7f8a540692a0 2024-03-11T22:59:55.157 INFO:teuthology.orchestra.run.smithi153.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-11T22:59:55.157 INFO:teuthology.orchestra.run.smithi153.stdout:smithi153 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300MV400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-11T22:59:55.157 INFO:teuthology.orchestra.run.smithi153.stdout:smithi163 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300GU400BGN 372G No 73s ago Has a FileSystem, LVM detected 2024-03-11T22:59:55.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:55.155+0000 7f8a7101c700 1 -- 172.21.15.153:0/925456712 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f8a540692a0 msgr2=0x7f8a5406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:55.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:55.155+0000 7f8a7101c700 1 --2- 172.21.15.153:0/925456712 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f8a540692a0 0x7f8a5406b750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8a60007fd0 tx=0x7f8a60009040 comp rx=0 tx=0).stop 2024-03-11T22:59:55.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:55.155+0000 7f8a7101c700 1 -- 172.21.15.153:0/925456712 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a6c0fbe40 msgr2=0x7f8a6c10c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:55.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:55.155+0000 7f8a7101c700 1 --2- 172.21.15.153:0/925456712 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a6c0fbe40 0x7f8a6c10c830 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c0093b0 tx=0x7f8a5c009e30 comp rx=0 tx=0).stop 2024-03-11T22:59:55.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:55.155+0000 7f8a7101c700 1 -- 172.21.15.153:0/925456712 shutdown_connections 2024-03-11T22:59:55.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:55.155+0000 7f8a7101c700 1 --2- 172.21.15.153:0/925456712 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a6c0fbe40 0x7f8a6c10c830 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:55.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:55.155+0000 7f8a7101c700 1 --2- 172.21.15.153:0/925456712 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f8a540692a0 0x7f8a5406b750 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:55.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:55.155+0000 7f8a7101c700 1 --2- 172.21.15.153:0/925456712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a6c0fcfb0 0x7f8a6c10cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:55.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:55.155+0000 7f8a7101c700 1 -- 172.21.15.153:0/925456712 >> 172.21.15.153:0/925456712 conn(0x7f8a6c0f7520 msgr2=0x7f8a6c100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:55.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:55.155+0000 7f8a7101c700 1 -- 172.21.15.153:0/925456712 shutdown_connections 2024-03-11T22:59:55.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:55.155+0000 7f8a7101c700 1 -- 172.21.15.153:0/925456712 wait complete. 2024-03-11T22:59:55.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:55 smithi163 ceph-mon[35584]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:55.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:55 smithi153 ceph-mon[28962]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:55.842 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-11T22:59:55.853 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi153.front.sepia.ceph.com 2024-03-11T22:59:55.853 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-03-11T22:59:56.112 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T22:59:56.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:56 smithi153 ceph-mon[28962]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:59:56.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:56 smithi163 ceph-mon[35584]: from='client.14436 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:59:57.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:57.708+0000 7fc3535d1700 1 -- 172.21.15.153:0/2415107438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc34c0fbe40 msgr2=0x7fc34c0fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:57.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:57.708+0000 7fc3535d1700 1 --2- 172.21.15.153:0/2415107438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc34c0fbe40 0x7fc34c0fc250 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc344009ce0 tx=0x7fc344004d10 comp rx=0 tx=0).stop 2024-03-11T22:59:57.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:57.708+0000 7fc3535d1700 1 -- 172.21.15.153:0/2415107438 shutdown_connections 2024-03-11T22:59:57.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:57.708+0000 7fc3535d1700 1 --2- 172.21.15.153:0/2415107438 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc34c0fcfb0 0x7fc34c0fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:57.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:57.708+0000 7fc3535d1700 1 --2- 172.21.15.153:0/2415107438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc34c0fbe40 0x7fc34c0fc250 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:57.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:57.708+0000 7fc3535d1700 1 -- 172.21.15.153:0/2415107438 >> 172.21.15.153:0/2415107438 conn(0x7fc34c0f74e0 msgr2=0x7fc34c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:57.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:57.708+0000 7fc3535d1700 1 -- 172.21.15.153:0/2415107438 shutdown_connections 2024-03-11T22:59:57.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:57.708+0000 7fc3535d1700 1 -- 172.21.15.153:0/2415107438 wait complete. 2024-03-11T22:59:57.713 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:57 smithi153 ceph-mon[28962]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:57.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:57.708+0000 7fc3535d1700 1 Processor -- start 2024-03-11T22:59:57.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:57.709+0000 7fc3535d1700 1 -- start start 2024-03-11T22:59:57.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:57.709+0000 7fc3535d1700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc34c0fbe40 0x7fc34c0fff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:57.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:57.709+0000 7fc3535d1700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc34c0fcfb0 0x7fc34c0fe380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:57.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:57.709+0000 7fc3535d1700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc34c0fe980 con 0x7fc34c0fbe40 2024-03-11T22:59:57.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:57.709+0000 7fc3535d1700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc34c0feaf0 con 0x7fc34c0fcfb0 2024-03-11T22:59:57.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:57.709+0000 7fc35136d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc34c0fbe40 0x7fc34c0fff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:57.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:57.709+0000 7fc35136d700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc34c0fbe40 0x7fc34c0fff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53974/0 (socket says 172.21.15.153:53974) 2024-03-11T22:59:57.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:57.709+0000 7fc35136d700 1 -- 172.21.15.153:0/3203039295 learned_addr learned my addr 172.21.15.153:0/3203039295 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T22:59:57.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:57.709+0000 7fc350b6c700 1 --2- 172.21.15.153:0/3203039295 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc34c0fcfb0 0x7fc34c0fe380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:57.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:57.709+0000 7fc35136d700 1 -- 172.21.15.153:0/3203039295 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc34c0fcfb0 msgr2=0x7fc34c0fe380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:57.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:57.709+0000 7fc35136d700 1 --2- 172.21.15.153:0/3203039295 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc34c0fcfb0 0x7fc34c0fe380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:57.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:57.709+0000 7fc35136d700 1 -- 172.21.15.153:0/3203039295 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc344009940 con 0x7fc34c0fbe40 2024-03-11T22:59:57.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:57.709+0000 7fc35136d700 1 --2- 172.21.15.153:0/3203039295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc34c0fbe40 0x7fc34c0fff00 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc344015040 tx=0x7fc344004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:57.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:57.710+0000 7fc341ffb700 1 -- 172.21.15.153:0/3203039295 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc34400e040 con 0x7fc34c0fbe40 2024-03-11T22:59:57.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:57.710+0000 7fc3535d1700 1 -- 172.21.15.153:0/3203039295 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc34c0fed70 con 0x7fc34c0fbe40 2024-03-11T22:59:57.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:57.710+0000 7fc341ffb700 1 -- 172.21.15.153:0/3203039295 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc3440128e0 con 0x7fc34c0fbe40 2024-03-11T22:59:57.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:57.710+0000 7fc341ffb700 1 -- 172.21.15.153:0/3203039295 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc34401c840 con 0x7fc34c0fbe40 2024-03-11T22:59:57.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:57.710+0000 7fc3535d1700 1 -- 172.21.15.153:0/3203039295 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc34c115ad0 con 0x7fc34c0fbe40 2024-03-11T22:59:57.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:57.711+0000 7fc3535d1700 1 -- 172.21.15.153:0/3203039295 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc34c04bed0 con 0x7fc34c0fbe40 2024-03-11T22:59:57.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:57.711+0000 7fc341ffb700 1 -- 172.21.15.153:0/3203039295 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fc344012a50 con 0x7fc34c0fbe40 2024-03-11T22:59:57.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:57.711+0000 7fc341ffb700 1 --2- 172.21.15.153:0/3203039295 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fc3380692a0 0x7fc33806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T22:59:57.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:57.712+0000 7fc341ffb700 1 -- 172.21.15.153:0/3203039295 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc344088b20 con 0x7fc34c0fbe40 2024-03-11T22:59:57.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:57.712+0000 7fc350b6c700 1 --2- 172.21.15.153:0/3203039295 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fc3380692a0 0x7fc33806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T22:59:57.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:57.712+0000 7fc350b6c700 1 --2- 172.21.15.153:0/3203039295 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fc3380692a0 0x7fc33806b750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc33c004d60 tx=0x7fc33c00b040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T22:59:57.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:57.716+0000 7fc341ffb700 1 -- 172.21.15.153:0/3203039295 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7fc344058ba0 con 0x7fc34c0fbe40 2024-03-11T22:59:57.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:57 smithi163 ceph-mon[35584]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:57.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:57.924+0000 7fc3535d1700 1 -- 172.21.15.153:0/3203039295 --> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fc34c065dd0 con 0x7fc3380692a0 2024-03-11T22:59:58.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:58 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-03-11T22:59:58.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:58 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-03-11T22:59:59.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:59.476+0000 7fc341ffb700 1 -- 172.21.15.153:0/3203039295 <== mgr.14204 v2:172.21.15.153:6800/29938766 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc34c065dd0 con 0x7fc3380692a0 2024-03-11T22:59:59.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:59.482+0000 7fc3535d1700 1 -- 172.21.15.153:0/3203039295 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fc3380692a0 msgr2=0x7fc33806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:59.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:59.482+0000 7fc3535d1700 1 --2- 172.21.15.153:0/3203039295 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fc3380692a0 0x7fc33806b750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc33c004d60 tx=0x7fc33c00b040 comp rx=0 tx=0).stop 2024-03-11T22:59:59.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:59.482+0000 7fc3535d1700 1 -- 172.21.15.153:0/3203039295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc34c0fbe40 msgr2=0x7fc34c0fff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T22:59:59.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:59.482+0000 7fc3535d1700 1 --2- 172.21.15.153:0/3203039295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc34c0fbe40 0x7fc34c0fff00 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc344015040 tx=0x7fc344004d10 comp rx=0 tx=0).stop 2024-03-11T22:59:59.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:59.482+0000 7fc3535d1700 1 -- 172.21.15.153:0/3203039295 shutdown_connections 2024-03-11T22:59:59.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:59.482+0000 7fc3535d1700 1 --2- 172.21.15.153:0/3203039295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc34c0fbe40 0x7fc34c0fff00 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:59.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:59.482+0000 7fc3535d1700 1 --2- 172.21.15.153:0/3203039295 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fc3380692a0 0x7fc33806b750 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:59.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:59.482+0000 7fc3535d1700 1 --2- 172.21.15.153:0/3203039295 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc34c0fcfb0 0x7fc34c0fe380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T22:59:59.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:59.482+0000 7fc3535d1700 1 -- 172.21.15.153:0/3203039295 >> 172.21.15.153:0/3203039295 conn(0x7fc34c0f74e0 msgr2=0x7fc34c0656c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T22:59:59.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:59.482+0000 7fc3535d1700 1 -- 172.21.15.153:0/3203039295 shutdown_connections 2024-03-11T22:59:59.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T22:59:59.482+0000 7fc3535d1700 1 -- 172.21.15.153:0/3203039295 wait complete. 2024-03-11T22:59:59.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:59 smithi163 ceph-mon[35584]: from='client.14440 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:59:59.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:59 smithi163 ceph-mon[35584]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:59.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:59 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-03-11T22:59:59.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:59 smithi163 ceph-mon[35584]: osdmap e42: 6 total, 6 up, 6 in 2024-03-11T22:59:59.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 22:59:59 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-03-11T22:59:59.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:59 smithi153 ceph-mon[28962]: from='client.14440 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:59:59.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:59 smithi153 ceph-mon[28962]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T22:59:59.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:59 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-03-11T22:59:59.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:59 smithi153 ceph-mon[28962]: osdmap e42: 6 total, 6 up, 6 in 2024-03-11T22:59:59.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:59 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-03-11T22:59:59.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 22:59:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T22:59:59.459+0000 7f0d9f420700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-11T23:00:00.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T22:59:59.999+0000 7f0da1c25700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 1 filesystem is offline; 1 filesystem is online with fewer MDS than max_mds 2024-03-11T23:00:00.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T22:59:59.999+0000 7f0da1c25700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-11T23:00:00.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T22:59:59.999+0000 7f0da1c25700 -1 log_channel(cluster) log [ERR] : fs cephfs is offline because no MDS is active for it. 2024-03-11T23:00:00.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T22:59:59.999+0000 7f0da1c25700 -1 log_channel(cluster) log [ERR] : [WRN] MDS_UP_LESS_THAN_MAX: 1 filesystem is online with fewer MDS than max_mds 2024-03-11T23:00:00.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T22:59:59.999+0000 7f0da1c25700 -1 log_channel(cluster) log [ERR] : fs cephfs has 0 MDS online, but wants 1 2024-03-11T23:00:00.375 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-11T23:00:00.719 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:00 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-03-11T23:00:00.719 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:00 smithi153 ceph-mon[28962]: osdmap e43: 6 total, 6 up, 6 in 2024-03-11T23:00:00.719 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:00 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-03-11T23:00:00.720 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:00 smithi153 ceph-mon[28962]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-11T23:00:00.720 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:00 smithi153 ceph-mon[28962]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-11T23:00:00.720 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:00 smithi153 ceph-mon[28962]: osdmap e44: 6 total, 6 up, 6 in 2024-03-11T23:00:00.720 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:00 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-03-11T23:00:00.720 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:00 smithi153 ceph-mon[28962]: fsmap cephfs:0 2024-03-11T23:00:00.720 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:00 smithi153 ceph-mon[28962]: Saving service mds.cephfs spec with placement count:4 2024-03-11T23:00:00.720 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:00 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:00.720 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:00 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:00:00.720 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:00 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:00:00.720 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:00 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:00:00.720 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:00 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:00.720 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:00 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.qxjyxz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T23:00:00.720 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:00 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.qxjyxz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T23:00:00.720 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:00 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:00:00.721 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:00 smithi153 ceph-mon[28962]: Deploying daemon mds.cephfs.smithi153.qxjyxz on smithi153 2024-03-11T23:00:00.721 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:00 smithi153 ceph-mon[28962]: Health detail: HEALTH_ERR 1 filesystem is offline; 1 filesystem is online with fewer MDS than max_mds 2024-03-11T23:00:00.721 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:00 smithi153 ceph-mon[28962]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-11T23:00:00.721 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:00 smithi153 ceph-mon[28962]: fs cephfs is offline because no MDS is active for it. 2024-03-11T23:00:00.721 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:00 smithi153 ceph-mon[28962]: [WRN] MDS_UP_LESS_THAN_MAX: 1 filesystem is online with fewer MDS than max_mds 2024-03-11T23:00:00.721 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:00 smithi153 ceph-mon[28962]: fs cephfs has 0 MDS online, but wants 1 2024-03-11T23:00:00.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:00 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-03-11T23:00:00.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:00 smithi163 ceph-mon[35584]: osdmap e43: 6 total, 6 up, 6 in 2024-03-11T23:00:00.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:00 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-03-11T23:00:00.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:00 smithi163 ceph-mon[35584]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-11T23:00:00.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:00 smithi163 ceph-mon[35584]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-11T23:00:00.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:00 smithi163 ceph-mon[35584]: osdmap e44: 6 total, 6 up, 6 in 2024-03-11T23:00:00.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:00 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-03-11T23:00:00.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:00 smithi163 ceph-mon[35584]: fsmap cephfs:0 2024-03-11T23:00:00.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:00 smithi163 ceph-mon[35584]: Saving service mds.cephfs spec with placement count:4 2024-03-11T23:00:00.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:00 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:00.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:00 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:00:00.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:00 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:00:00.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:00 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:00:00.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:00 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:00.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:00 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.qxjyxz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T23:00:00.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:00 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.qxjyxz", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T23:00:00.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:00 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:00:00.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:00 smithi163 ceph-mon[35584]: Deploying daemon mds.cephfs.smithi153.qxjyxz on smithi153 2024-03-11T23:00:00.740 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:00 smithi163 ceph-mon[35584]: Health detail: HEALTH_ERR 1 filesystem is offline; 1 filesystem is online with fewer MDS than max_mds 2024-03-11T23:00:00.740 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:00 smithi163 ceph-mon[35584]: [ERR] MDS_ALL_DOWN: 1 filesystem is offline 2024-03-11T23:00:00.740 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:00 smithi163 ceph-mon[35584]: fs cephfs is offline because no MDS is active for it. 2024-03-11T23:00:00.740 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:00 smithi163 ceph-mon[35584]: [WRN] MDS_UP_LESS_THAN_MAX: 1 filesystem is online with fewer MDS than max_mds 2024-03-11T23:00:00.740 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:00 smithi163 ceph-mon[35584]: fs cephfs has 0 MDS online, but wants 1 2024-03-11T23:00:01.159 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T23:00:01.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:01 smithi153 ceph-mon[28962]: pgmap v173: 65 pgs: 13 creating+peering, 4 active+clean, 48 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T23:00:01.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:01 smithi153 ceph-mon[28962]: osdmap e45: 6 total, 6 up, 6 in 2024-03-11T23:00:01.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:01 smithi163 ceph-mon[35584]: pgmap v173: 65 pgs: 13 creating+peering, 4 active+clean, 48 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T23:00:01.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:01 smithi163 ceph-mon[35584]: osdmap e45: 6 total, 6 up, 6 in 2024-03-11T23:00:02.721 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:02 smithi153 ceph-mon[28962]: osdmap e46: 6 total, 6 up, 6 in 2024-03-11T23:00:02.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:02 smithi163 ceph-mon[35584]: osdmap e46: 6 total, 6 up, 6 in 2024-03-11T23:00:03.668 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:03 smithi153 ceph-mon[28962]: pgmap v176: 65 pgs: 17 creating+peering, 28 active+clean, 20 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T23:00:03.669 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:03 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:03.669 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:03 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:00:03.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:03 smithi163 ceph-mon[35584]: pgmap v176: 65 pgs: 17 creating+peering, 28 active+clean, 20 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T23:00:03.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:03 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:03.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:03 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:00:03.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:03.765+0000 7f54f051b700 1 -- 172.21.15.153:0/4257107302 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54e80fda80 msgr2=0x7f54e80ffe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:03.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:03.765+0000 7f54f051b700 1 --2- 172.21.15.153:0/4257107302 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54e80fda80 0x7f54e80ffe60 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f54e4004940 tx=0x7f54e4009e30 comp rx=0 tx=0).stop 2024-03-11T23:00:03.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:03.765+0000 7f54f051b700 1 -- 172.21.15.153:0/4257107302 shutdown_connections 2024-03-11T23:00:03.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:03.765+0000 7f54f051b700 1 --2- 172.21.15.153:0/4257107302 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54e81003a0 0x7f54e8102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:03.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:03.765+0000 7f54f051b700 1 --2- 172.21.15.153:0/4257107302 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54e80fda80 0x7f54e80ffe60 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:03.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:03.765+0000 7f54f051b700 1 -- 172.21.15.153:0/4257107302 >> 172.21.15.153:0/4257107302 conn(0x7f54e80f7500 msgr2=0x7f54e80f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:00:03.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:03.765+0000 7f54f051b700 1 -- 172.21.15.153:0/4257107302 shutdown_connections 2024-03-11T23:00:03.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:03.765+0000 7f54f051b700 1 -- 172.21.15.153:0/4257107302 wait complete. 2024-03-11T23:00:03.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:03.766+0000 7f54f051b700 1 Processor -- start 2024-03-11T23:00:03.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:03.766+0000 7f54f051b700 1 -- start start 2024-03-11T23:00:03.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:03.766+0000 7f54f051b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54e80fda80 0x7f54e80fd510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:03.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:03.766+0000 7f54f051b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54e81003a0 0x7f54e80fb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:03.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:03.766+0000 7f54f051b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e80fbf90 con 0x7f54e80fda80 2024-03-11T23:00:03.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:03.766+0000 7f54f051b700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e80fc100 con 0x7f54e81003a0 2024-03-11T23:00:03.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:03.766+0000 7f54ee2b7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54e80fda80 0x7f54e80fd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:03.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:03.766+0000 7f54ee2b7700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54e80fda80 0x7f54e80fd510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35586/0 (socket says 172.21.15.153:35586) 2024-03-11T23:00:03.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:03.766+0000 7f54ee2b7700 1 -- 172.21.15.153:0/3222959012 learned_addr learned my addr 172.21.15.153:0/3222959012 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T23:00:03.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:03.766+0000 7f54edab6700 1 --2- 172.21.15.153:0/3222959012 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54e81003a0 0x7f54e80fb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:03.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:03.766+0000 7f54ee2b7700 1 -- 172.21.15.153:0/3222959012 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54e81003a0 msgr2=0x7f54e80fb990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:03.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:03.766+0000 7f54ee2b7700 1 --2- 172.21.15.153:0/3222959012 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54e81003a0 0x7f54e80fb990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:03.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:03.766+0000 7f54ee2b7700 1 -- 172.21.15.153:0/3222959012 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e4005040 con 0x7f54e80fda80 2024-03-11T23:00:03.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:03.767+0000 7f54ee2b7700 1 --2- 172.21.15.153:0/3222959012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54e80fda80 0x7f54e80fd510 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f54e40093b0 tx=0x7f54e4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:00:03.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:03.767+0000 7f54deffd700 1 -- 172.21.15.153:0/3222959012 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54e400e040 con 0x7f54e80fda80 2024-03-11T23:00:03.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:03.767+0000 7f54deffd700 1 -- 172.21.15.153:0/3222959012 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f54e400bcc0 con 0x7f54e80fda80 2024-03-11T23:00:03.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:03.767+0000 7f54f051b700 1 -- 172.21.15.153:0/3222959012 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54e80fc380 con 0x7f54e80fda80 2024-03-11T23:00:03.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:03.767+0000 7f54deffd700 1 -- 172.21.15.153:0/3222959012 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f54e401c630 con 0x7f54e80fda80 2024-03-11T23:00:03.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:03.767+0000 7f54f051b700 1 -- 172.21.15.153:0/3222959012 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54e81158c0 con 0x7f54e80fda80 2024-03-11T23:00:03.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:03.768+0000 7f54f051b700 1 -- 172.21.15.153:0/3222959012 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54e8107a50 con 0x7f54e80fda80 2024-03-11T23:00:03.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:03.768+0000 7f54deffd700 1 -- 172.21.15.153:0/3222959012 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f54e4013980 con 0x7f54e80fda80 2024-03-11T23:00:03.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:03.769+0000 7f54deffd700 1 --2- 172.21.15.153:0/3222959012 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f54d40692a0 0x7f54d406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:03.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:03.769+0000 7f54edab6700 1 --2- 172.21.15.153:0/3222959012 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f54d40692a0 0x7f54d406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:03.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:03.769+0000 7f54deffd700 1 -- 172.21.15.153:0/3222959012 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f54e4088430 con 0x7f54e80fda80 2024-03-11T23:00:03.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:03.769+0000 7f54edab6700 1 --2- 172.21.15.153:0/3222959012 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f54d40692a0 0x7f54d406b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f54d8009fd0 tx=0x7f54d800b040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:00:03.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:03.773+0000 7f54deffd700 1 -- 172.21.15.153:0/3222959012 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f54e4058290 con 0x7f54e80fda80 2024-03-11T23:00:04.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:04.001+0000 7f54f051b700 1 -- 172.21.15.153:0/3222959012 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f54e802b3a0 con 0x7f54e80fda80 2024-03-11T23:00:04.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:04.001+0000 7f54deffd700 1 -- 172.21.15.153:0/3222959012 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1076 (secure 0 0 0) 0x7f54e4057e20 con 0x7f54e80fda80 2024-03-11T23:00:04.006 INFO:teuthology.orchestra.run.smithi153.stdout:e2 2024-03-11T23:00:04.006 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T23:00:04.006 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-11T23:00:04.006 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-03-11T23:00:04.006 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T23:00:04.006 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-03-11T23:00:04.006 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-03-11T23:00:04.006 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 2 2024-03-11T23:00:04.006 INFO:teuthology.orchestra.run.smithi153.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-11T23:00:04.007 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-03-11T22:59:59.460410+0000 2024-03-11T23:00:04.007 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-03-11T22:59:59.460465+0000 2024-03-11T23:00:04.007 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-03-11T23:00:04.007 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-03-11T23:00:04.007 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-03-11T23:00:04.007 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-03-11T23:00:04.007 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-03-11T23:00:04.007 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-03-11T23:00:04.007 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-03-11T23:00:04.007 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 0 2024-03-11T23:00:04.007 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-11T23:00:04.007 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 1 2024-03-11T23:00:04.007 INFO:teuthology.orchestra.run.smithi153.stdout:in 2024-03-11T23:00:04.007 INFO:teuthology.orchestra.run.smithi153.stdout:up {} 2024-03-11T23:00:04.007 INFO:teuthology.orchestra.run.smithi153.stdout:failed 2024-03-11T23:00:04.008 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-03-11T23:00:04.008 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-03-11T23:00:04.008 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-03-11T23:00:04.008 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-03-11T23:00:04.008 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data disabled 2024-03-11T23:00:04.008 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-03-11T23:00:04.008 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 0 2024-03-11T23:00:04.008 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T23:00:04.008 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T23:00:04.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:04.003+0000 7f54f051b700 1 -- 172.21.15.153:0/3222959012 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f54d40692a0 msgr2=0x7f54d406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:04.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:04.003+0000 7f54f051b700 1 --2- 172.21.15.153:0/3222959012 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f54d40692a0 0x7f54d406b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f54d8009fd0 tx=0x7f54d800b040 comp rx=0 tx=0).stop 2024-03-11T23:00:04.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:04.003+0000 7f54f051b700 1 -- 172.21.15.153:0/3222959012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54e80fda80 msgr2=0x7f54e80fd510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:04.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:04.003+0000 7f54f051b700 1 --2- 172.21.15.153:0/3222959012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54e80fda80 0x7f54e80fd510 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f54e40093b0 tx=0x7f54e4009e30 comp rx=0 tx=0).stop 2024-03-11T23:00:04.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:04.004+0000 7f54f051b700 1 -- 172.21.15.153:0/3222959012 shutdown_connections 2024-03-11T23:00:04.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:04.004+0000 7f54f051b700 1 --2- 172.21.15.153:0/3222959012 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54e80fda80 0x7f54e80fd510 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:04.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:04.004+0000 7f54f051b700 1 --2- 172.21.15.153:0/3222959012 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f54d40692a0 0x7f54d406b750 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:04.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:04.004+0000 7f54f051b700 1 --2- 172.21.15.153:0/3222959012 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54e81003a0 0x7f54e80fb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:04.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:04.004+0000 7f54f051b700 1 -- 172.21.15.153:0/3222959012 >> 172.21.15.153:0/3222959012 conn(0x7f54e80f7500 msgr2=0x7f54e80f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:00:04.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:04.004+0000 7f54f051b700 1 -- 172.21.15.153:0/3222959012 shutdown_connections 2024-03-11T23:00:04.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:04.004+0000 7f54f051b700 1 -- 172.21.15.153:0/3222959012 wait complete. 2024-03-11T23:00:04.010 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 2 2024-03-11T23:00:04.648 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:04 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:04.648 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:04 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:04.648 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:04 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:04.649 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:04 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi163.pmppga", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T23:00:04.649 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:04 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi163.pmppga", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T23:00:04.649 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:04 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:00:04.649 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:04 smithi153 ceph-mon[28962]: Deploying daemon mds.cephfs.smithi163.pmppga on smithi163 2024-03-11T23:00:04.649 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:04 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/3222959012' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-11T23:00:04.649 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:04 smithi153 ceph-mon[28962]: mds.? [v2:172.21.15.153:6826/633795058,v1:172.21.15.153:6827/633795058] up:boot 2024-03-11T23:00:04.649 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:04 smithi153 ceph-mon[28962]: daemon mds.cephfs.smithi153.qxjyxz assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-03-11T23:00:04.649 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:04 smithi153 ceph-mon[28962]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-11T23:00:04.649 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:04 smithi153 ceph-mon[28962]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-11T23:00:04.649 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:04 smithi153 ceph-mon[28962]: Cluster is now healthy 2024-03-11T23:00:04.649 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:04 smithi153 ceph-mon[28962]: fsmap cephfs:0 1 up:standby 2024-03-11T23:00:04.649 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:04 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.qxjyxz"}]: dispatch 2024-03-11T23:00:04.649 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:04 smithi153 ceph-mon[28962]: fsmap cephfs:1 {0=cephfs.smithi153.qxjyxz=up:creating} 2024-03-11T23:00:04.649 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:04 smithi153 ceph-mon[28962]: daemon mds.cephfs.smithi153.qxjyxz is now active in filesystem cephfs as rank 0 2024-03-11T23:00:04.672 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-11T23:00:04.683 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi153.front.sepia.ceph.com 2024-03-11T23:00:04.683 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- bash -c 'ceph fs set cephfs max_mds 2' 2024-03-11T23:00:04.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:04 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:04.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:04 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:04.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:04 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:04.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:04 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi163.pmppga", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T23:00:04.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:04 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi163.pmppga", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T23:00:04.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:04 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:00:04.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:04 smithi163 ceph-mon[35584]: Deploying daemon mds.cephfs.smithi163.pmppga on smithi163 2024-03-11T23:00:04.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:04 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/3222959012' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-11T23:00:04.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:04 smithi163 ceph-mon[35584]: mds.? [v2:172.21.15.153:6826/633795058,v1:172.21.15.153:6827/633795058] up:boot 2024-03-11T23:00:04.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:04 smithi163 ceph-mon[35584]: daemon mds.cephfs.smithi153.qxjyxz assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-03-11T23:00:04.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:04 smithi163 ceph-mon[35584]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-11T23:00:04.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:04 smithi163 ceph-mon[35584]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-11T23:00:04.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:04 smithi163 ceph-mon[35584]: Cluster is now healthy 2024-03-11T23:00:04.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:04 smithi163 ceph-mon[35584]: fsmap cephfs:0 1 up:standby 2024-03-11T23:00:04.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:04 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.qxjyxz"}]: dispatch 2024-03-11T23:00:04.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:04 smithi163 ceph-mon[35584]: fsmap cephfs:1 {0=cephfs.smithi153.qxjyxz=up:creating} 2024-03-11T23:00:04.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:04 smithi163 ceph-mon[35584]: daemon mds.cephfs.smithi153.qxjyxz is now active in filesystem cephfs as rank 0 2024-03-11T23:00:04.945 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T23:00:05.840 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:05 smithi163 ceph-mon[35584]: pgmap v177: 65 pgs: 10 creating+peering, 55 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T23:00:05.841 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:05 smithi163 ceph-mon[35584]: mds.? [v2:172.21.15.153:6826/633795058,v1:172.21.15.153:6827/633795058] up:active 2024-03-11T23:00:05.841 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:05 smithi163 ceph-mon[35584]: fsmap cephfs:1 {0=cephfs.smithi153.qxjyxz=up:active} 2024-03-11T23:00:06.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:05 smithi153 ceph-mon[28962]: pgmap v177: 65 pgs: 10 creating+peering, 55 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-11T23:00:06.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:05 smithi153 ceph-mon[28962]: mds.? [v2:172.21.15.153:6826/633795058,v1:172.21.15.153:6827/633795058] up:active 2024-03-11T23:00:06.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:05 smithi153 ceph-mon[28962]: fsmap cephfs:1 {0=cephfs.smithi153.qxjyxz=up:active} 2024-03-11T23:00:06.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:06.664+0000 7f0c3ffa8700 1 -- 172.21.15.153:0/1418445244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c381000d0 msgr2=0x7f0c38102530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:06.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:06.664+0000 7f0c3ffa8700 1 --2- 172.21.15.153:0/1418445244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c381000d0 0x7f0c38102530 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f0c28004940 tx=0x7f0c28009e30 comp rx=0 tx=0).stop 2024-03-11T23:00:06.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:06.664+0000 7f0c3ffa8700 1 -- 172.21.15.153:0/1418445244 shutdown_connections 2024-03-11T23:00:06.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:06.664+0000 7f0c3ffa8700 1 --2- 172.21.15.153:0/1418445244 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c381000d0 0x7f0c38102530 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:06.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:06.664+0000 7f0c3ffa8700 1 --2- 172.21.15.153:0/1418445244 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0c38066c70 0x7f0c380ffb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:06.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:06.664+0000 7f0c3ffa8700 1 -- 172.21.15.153:0/1418445244 >> 172.21.15.153:0/1418445244 conn(0x7f0c380f7510 msgr2=0x7f0c380f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:00:06.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:06.664+0000 7f0c3ffa8700 1 -- 172.21.15.153:0/1418445244 shutdown_connections 2024-03-11T23:00:06.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:06.664+0000 7f0c3ffa8700 1 -- 172.21.15.153:0/1418445244 wait complete. 2024-03-11T23:00:06.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:06.664+0000 7f0c3ffa8700 1 Processor -- start 2024-03-11T23:00:06.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:06.664+0000 7f0c3ffa8700 1 -- start start 2024-03-11T23:00:06.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:06.665+0000 7f0c3ffa8700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0c38066c70 0x7f0c3810d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:06.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:06.665+0000 7f0c3ffa8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c381000d0 0x7f0c3810de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:06.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:06.665+0000 7f0c3ffa8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3810e460 con 0x7f0c381000d0 2024-03-11T23:00:06.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:06.665+0000 7f0c3dd44700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0c38066c70 0x7f0c3810d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:06.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:06.665+0000 7f0c3dd44700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0c38066c70 0x7f0c3810d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.153:49044/0 (socket says 172.21.15.153:49044) 2024-03-11T23:00:06.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:06.665+0000 7f0c3ffa8700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3810ed00 con 0x7f0c38066c70 2024-03-11T23:00:06.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:06.665+0000 7f0c3dd44700 1 -- 172.21.15.153:0/3452228100 learned_addr learned my addr 172.21.15.153:0/3452228100 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T23:00:06.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:06.665+0000 7f0c3d543700 1 --2- 172.21.15.153:0/3452228100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c381000d0 0x7f0c3810de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:06.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:06.666+0000 7f0c3dd44700 1 -- 172.21.15.153:0/3452228100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c381000d0 msgr2=0x7f0c3810de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:06.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:06.666+0000 7f0c3dd44700 1 --2- 172.21.15.153:0/3452228100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c381000d0 0x7f0c3810de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:06.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:06.666+0000 7f0c3dd44700 1 -- 172.21.15.153:0/3452228100 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c28005040 con 0x7f0c38066c70 2024-03-11T23:00:06.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:06.666+0000 7f0c3dd44700 1 --2- 172.21.15.153:0/3452228100 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0c38066c70 0x7f0c3810d920 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400eb40 tx=0x7f0c3400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:00:06.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:06.666+0000 7f0c2e7fc700 1 -- 172.21.15.153:0/3452228100 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c34007cc0 con 0x7f0c38066c70 2024-03-11T23:00:06.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:06.666+0000 7f0c2e7fc700 1 -- 172.21.15.153:0/3452228100 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0c3400f040 con 0x7f0c38066c70 2024-03-11T23:00:06.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:06.666+0000 7f0c3ffa8700 1 -- 172.21.15.153:0/3452228100 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c3810ef80 con 0x7f0c38066c70 2024-03-11T23:00:06.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:06.666+0000 7f0c2e7fc700 1 -- 172.21.15.153:0/3452228100 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c3400ca10 con 0x7f0c38066c70 2024-03-11T23:00:06.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:06.666+0000 7f0c3ffa8700 1 -- 172.21.15.153:0/3452228100 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c3810f4c0 con 0x7f0c38066c70 2024-03-11T23:00:06.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:06.667+0000 7f0c2e7fc700 1 -- 172.21.15.153:0/3452228100 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f0c3400cc80 con 0x7f0c38066c70 2024-03-11T23:00:06.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:06.668+0000 7f0c3ffa8700 1 -- 172.21.15.153:0/3452228100 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c38107a60 con 0x7f0c38066c70 2024-03-11T23:00:06.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:06.668+0000 7f0c2e7fc700 1 --2- 172.21.15.153:0/3452228100 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f0c2406d6b0 0x7f0c2406fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:06.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:06.668+0000 7f0c2e7fc700 1 -- 172.21.15.153:0/3452228100 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f0c34086e00 con 0x7f0c38066c70 2024-03-11T23:00:06.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:06.668+0000 7f0c3d543700 1 --2- 172.21.15.153:0/3452228100 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f0c2406d6b0 0x7f0c2406fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:06.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:06.668+0000 7f0c3d543700 1 --2- 172.21.15.153:0/3452228100 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f0c2406d6b0 0x7f0c2406fb60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0c28000c00 tx=0x7f0c28009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:00:06.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:06.673+0000 7f0c2e7fc700 1 -- 172.21.15.153:0/3452228100 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f0c3404dd20 con 0x7f0c38066c70 2024-03-11T23:00:06.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:06.920+0000 7f0c3ffa8700 1 -- 172.21.15.153:0/3452228100 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f0c3802b2a0 con 0x7f0c38066c70 2024-03-11T23:00:07.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:07 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:07.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:07 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:07.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:07 smithi153 ceph-mon[28962]: pgmap v178: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-03-11T23:00:07.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:07 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:07.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:07 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.yuzdaj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T23:00:07.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:07 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.yuzdaj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T23:00:07.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:07 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:00:07.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:07 smithi153 ceph-mon[28962]: Deploying daemon mds.cephfs.smithi153.yuzdaj on smithi153 2024-03-11T23:00:07.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:07 smithi153 ceph-mon[28962]: mds.? [v2:172.21.15.163:6824/3959509309,v1:172.21.15.163:6825/3959509309] up:boot 2024-03-11T23:00:07.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:07 smithi153 ceph-mon[28962]: fsmap cephfs:1 {0=cephfs.smithi153.qxjyxz=up:active} 1 up:standby 2024-03-11T23:00:07.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:07 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi163.pmppga"}]: dispatch 2024-03-11T23:00:07.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:07 smithi153 ceph-mon[28962]: fsmap cephfs:1 {0=cephfs.smithi153.qxjyxz=up:active} 1 up:standby 2024-03-11T23:00:07.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:07 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/3452228100' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-03-11T23:00:07.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:07 smithi153 ceph-mon[28962]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-03-11T23:00:07.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:07 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:07.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:07 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:07.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:07 smithi163 ceph-mon[35584]: pgmap v178: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s wr, 3 op/s 2024-03-11T23:00:07.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:07 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:07.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:07 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.yuzdaj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T23:00:07.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:07 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi153.yuzdaj", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T23:00:07.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:07 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:00:07.489 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:07 smithi163 ceph-mon[35584]: Deploying daemon mds.cephfs.smithi153.yuzdaj on smithi153 2024-03-11T23:00:07.489 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:07 smithi163 ceph-mon[35584]: mds.? [v2:172.21.15.163:6824/3959509309,v1:172.21.15.163:6825/3959509309] up:boot 2024-03-11T23:00:07.489 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:07 smithi163 ceph-mon[35584]: fsmap cephfs:1 {0=cephfs.smithi153.qxjyxz=up:active} 1 up:standby 2024-03-11T23:00:07.489 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:07 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi163.pmppga"}]: dispatch 2024-03-11T23:00:07.489 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:07 smithi163 ceph-mon[35584]: fsmap cephfs:1 {0=cephfs.smithi153.qxjyxz=up:active} 1 up:standby 2024-03-11T23:00:07.489 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:07 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/3452228100' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-03-11T23:00:07.489 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:07 smithi163 ceph-mon[35584]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]: dispatch 2024-03-11T23:00:07.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:07.588+0000 7f0c2e7fc700 1 -- 172.21.15.153:0/3452228100 <== mon.1 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v8) v1 ==== 105+0+0 (secure 0 0 0) 0x7f0c340095f0 con 0x7f0c38066c70 2024-03-11T23:00:07.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:07.591+0000 7f0c3ffa8700 1 -- 172.21.15.153:0/3452228100 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f0c2406d6b0 msgr2=0x7f0c2406fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:07.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:07.591+0000 7f0c3ffa8700 1 --2- 172.21.15.153:0/3452228100 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f0c2406d6b0 0x7f0c2406fb60 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0c28000c00 tx=0x7f0c28009e30 comp rx=0 tx=0).stop 2024-03-11T23:00:07.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:07.591+0000 7f0c3ffa8700 1 -- 172.21.15.153:0/3452228100 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0c38066c70 msgr2=0x7f0c3810d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:07.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:07.591+0000 7f0c3ffa8700 1 --2- 172.21.15.153:0/3452228100 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0c38066c70 0x7f0c3810d920 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400eb40 tx=0x7f0c3400b040 comp rx=0 tx=0).stop 2024-03-11T23:00:07.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:07.591+0000 7f0c3ffa8700 1 -- 172.21.15.153:0/3452228100 shutdown_connections 2024-03-11T23:00:07.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:07.591+0000 7f0c3ffa8700 1 --2- 172.21.15.153:0/3452228100 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f0c2406d6b0 0x7f0c2406fb60 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:07.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:07.591+0000 7f0c3ffa8700 1 --2- 172.21.15.153:0/3452228100 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0c38066c70 0x7f0c3810d920 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:07.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:07.591+0000 7f0c3ffa8700 1 --2- 172.21.15.153:0/3452228100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c381000d0 0x7f0c3810de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:07.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:07.591+0000 7f0c3ffa8700 1 -- 172.21.15.153:0/3452228100 >> 172.21.15.153:0/3452228100 conn(0x7f0c380f7510 msgr2=0x7f0c380f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:00:07.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:07.591+0000 7f0c3ffa8700 1 -- 172.21.15.153:0/3452228100 shutdown_connections 2024-03-11T23:00:07.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:07.591+0000 7f0c3ffa8700 1 -- 172.21.15.153:0/3452228100 wait complete. 2024-03-11T23:00:08.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:08 smithi153 ceph-mon[28962]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-11T23:00:08.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:08 smithi153 ceph-mon[28962]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-03-11T23:00:08.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:08 smithi153 ceph-mon[28962]: daemon mds.cephfs.smithi163.pmppga assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-11T23:00:08.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:08 smithi153 ceph-mon[28962]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-11T23:00:08.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:08 smithi153 ceph-mon[28962]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-11T23:00:08.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:08 smithi153 ceph-mon[28962]: fsmap cephfs:1 {0=cephfs.smithi153.qxjyxz=up:active} 1 up:standby 2024-03-11T23:00:08.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:08 smithi153 ceph-mon[28962]: fsmap cephfs:2 {0=cephfs.smithi153.qxjyxz=up:active,1=cephfs.smithi163.pmppga=up:creating} 2024-03-11T23:00:08.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:08 smithi153 ceph-mon[28962]: daemon mds.cephfs.smithi163.pmppga is now active in filesystem cephfs as rank 1 2024-03-11T23:00:08.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:08 smithi163 ceph-mon[35584]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-11T23:00:08.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:08 smithi163 ceph-mon[35584]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]': finished 2024-03-11T23:00:08.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:08 smithi163 ceph-mon[35584]: daemon mds.cephfs.smithi163.pmppga assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-11T23:00:08.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:08 smithi163 ceph-mon[35584]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-03-11T23:00:08.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:08 smithi163 ceph-mon[35584]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-11T23:00:08.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:08 smithi163 ceph-mon[35584]: fsmap cephfs:1 {0=cephfs.smithi153.qxjyxz=up:active} 1 up:standby 2024-03-11T23:00:08.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:08 smithi163 ceph-mon[35584]: fsmap cephfs:2 {0=cephfs.smithi153.qxjyxz=up:active,1=cephfs.smithi163.pmppga=up:creating} 2024-03-11T23:00:08.489 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:08 smithi163 ceph-mon[35584]: daemon mds.cephfs.smithi163.pmppga is now active in filesystem cephfs as rank 1 2024-03-11T23:00:08.833 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-11T23:00:08.843 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi153.front.sepia.ceph.com 2024-03-11T23:00:08.843 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- bash -c 'ceph fs set cephfs allow_standby_replay true' 2024-03-11T23:00:09.373 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T23:00:09.647 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:09 smithi153 ceph-mon[28962]: pgmap v179: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s wr, 3 op/s 2024-03-11T23:00:09.648 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:09 smithi153 ceph-mon[28962]: mds.? [v2:172.21.15.163:6824/3959509309,v1:172.21.15.163:6825/3959509309] up:active 2024-03-11T23:00:09.648 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:09 smithi153 ceph-mon[28962]: fsmap cephfs:2 {0=cephfs.smithi153.qxjyxz=up:active,1=cephfs.smithi163.pmppga=up:active} 2024-03-11T23:00:09.648 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:09 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:09.648 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:09 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:09.648 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:09 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:09.648 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:09 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi163.kzggfd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T23:00:09.648 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:09 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi163.kzggfd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T23:00:09.648 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:09 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:00:09.856 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:09 smithi163 ceph-mon[35584]: pgmap v179: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1023 B/s wr, 3 op/s 2024-03-11T23:00:09.857 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:09 smithi163 ceph-mon[35584]: mds.? [v2:172.21.15.163:6824/3959509309,v1:172.21.15.163:6825/3959509309] up:active 2024-03-11T23:00:09.857 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:09 smithi163 ceph-mon[35584]: fsmap cephfs:2 {0=cephfs.smithi153.qxjyxz=up:active,1=cephfs.smithi163.pmppga=up:active} 2024-03-11T23:00:09.857 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:09 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:09.857 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:09 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:09.857 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:09 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:09.857 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:09 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi163.kzggfd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-11T23:00:09.857 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:09 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi163.kzggfd", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-11T23:00:09.857 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:09 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:00:10.656 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:10 smithi153 ceph-mon[28962]: Deploying daemon mds.cephfs.smithi163.kzggfd on smithi163 2024-03-11T23:00:10.656 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:10 smithi153 ceph-mon[28962]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-11T23:00:10.656 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:10 smithi153 ceph-mon[28962]: Cluster is now healthy 2024-03-11T23:00:10.656 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:10 smithi153 ceph-mon[28962]: mds.? [v2:172.21.15.153:6828/2364288180,v1:172.21.15.153:6829/2364288180] up:boot 2024-03-11T23:00:10.656 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:10 smithi153 ceph-mon[28962]: fsmap cephfs:2 {0=cephfs.smithi153.qxjyxz=up:active,1=cephfs.smithi163.pmppga=up:active} 1 up:standby 2024-03-11T23:00:10.656 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:10 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.yuzdaj"}]: dispatch 2024-03-11T23:00:10.860 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:10 smithi163 ceph-mon[35584]: Deploying daemon mds.cephfs.smithi163.kzggfd on smithi163 2024-03-11T23:00:10.860 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:10 smithi163 ceph-mon[35584]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-03-11T23:00:10.860 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:10 smithi163 ceph-mon[35584]: Cluster is now healthy 2024-03-11T23:00:10.860 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:10 smithi163 ceph-mon[35584]: mds.? [v2:172.21.15.153:6828/2364288180,v1:172.21.15.153:6829/2364288180] up:boot 2024-03-11T23:00:10.860 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:10 smithi163 ceph-mon[35584]: fsmap cephfs:2 {0=cephfs.smithi153.qxjyxz=up:active,1=cephfs.smithi163.pmppga=up:active} 1 up:standby 2024-03-11T23:00:10.860 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:10 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi153.yuzdaj"}]: dispatch 2024-03-11T23:00:10.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:10.877+0000 7f4468e86700 1 -- 172.21.15.153:0/514457468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44640fef90 msgr2=0x7f44640fb0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:10.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:10.877+0000 7f4468e86700 1 --2- 172.21.15.153:0/514457468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44640fef90 0x7f44640fb0d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4454004940 tx=0x7f4454009e30 comp rx=0 tx=0).stop 2024-03-11T23:00:10.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:10.877+0000 7f4468e86700 1 -- 172.21.15.153:0/514457468 shutdown_connections 2024-03-11T23:00:10.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:10.877+0000 7f4468e86700 1 --2- 172.21.15.153:0/514457468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44640fef90 0x7f44640fb0d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:10.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:10.877+0000 7f4468e86700 1 --2- 172.21.15.153:0/514457468 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44640fe640 0x7f44640fea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:10.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:10.877+0000 7f4468e86700 1 -- 172.21.15.153:0/514457468 >> 172.21.15.153:0/514457468 conn(0x7f44640f6d20 msgr2=0x7f44640f9150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:00:10.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:10.877+0000 7f4468e86700 1 -- 172.21.15.153:0/514457468 shutdown_connections 2024-03-11T23:00:10.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:10.877+0000 7f4468e86700 1 -- 172.21.15.153:0/514457468 wait complete. 2024-03-11T23:00:10.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:10.877+0000 7f4468e86700 1 Processor -- start 2024-03-11T23:00:10.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:10.878+0000 7f4468e86700 1 -- start start 2024-03-11T23:00:10.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:10.878+0000 7f4468e86700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44640fe640 0x7f446410dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:10.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:10.878+0000 7f4468e86700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44640fef90 0x7f446410e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:10.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:10.878+0000 7f4468e86700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446410efd0 con 0x7f44640fef90 2024-03-11T23:00:10.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:10.878+0000 7f4468e86700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446410f140 con 0x7f44640fe640 2024-03-11T23:00:10.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:10.878+0000 7f4461d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44640fef90 0x7f446410e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:10.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:10.878+0000 7f4461d9b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44640fef90 0x7f446410e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:35642/0 (socket says 172.21.15.153:35642) 2024-03-11T23:00:10.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:10.878+0000 7f4461d9b700 1 -- 172.21.15.153:0/660840369 learned_addr learned my addr 172.21.15.153:0/660840369 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T23:00:10.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:10.878+0000 7f4461d9b700 1 -- 172.21.15.153:0/660840369 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44640fe640 msgr2=0x7f446410dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:10.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:10.878+0000 7f4461d9b700 1 --2- 172.21.15.153:0/660840369 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44640fe640 0x7f446410dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:10.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:10.878+0000 7f4461d9b700 1 -- 172.21.15.153:0/660840369 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4454005040 con 0x7f44640fef90 2024-03-11T23:00:10.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:10.878+0000 7f4461d9b700 1 --2- 172.21.15.153:0/660840369 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44640fef90 0x7f446410e270 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4454009fd0 tx=0x7f4454009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:00:10.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:10.878+0000 7f4452ffd700 1 -- 172.21.15.153:0/660840369 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f445400e040 con 0x7f44640fef90 2024-03-11T23:00:10.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:10.878+0000 7f4452ffd700 1 -- 172.21.15.153:0/660840369 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f445400bcc0 con 0x7f44640fef90 2024-03-11T23:00:10.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:10.878+0000 7f4468e86700 1 -- 172.21.15.153:0/660840369 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f446410f3c0 con 0x7f44640fef90 2024-03-11T23:00:10.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:10.879+0000 7f4452ffd700 1 -- 172.21.15.153:0/660840369 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f445401c630 con 0x7f44640fef90 2024-03-11T23:00:10.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:10.879+0000 7f4468e86700 1 -- 172.21.15.153:0/660840369 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f446410f900 con 0x7f44640fef90 2024-03-11T23:00:10.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:10.879+0000 7f4468e86700 1 -- 172.21.15.153:0/660840369 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4464107b10 con 0x7f44640fef90 2024-03-11T23:00:10.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:10.880+0000 7f4452ffd700 1 -- 172.21.15.153:0/660840369 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f4454013980 con 0x7f44640fef90 2024-03-11T23:00:10.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:10.880+0000 7f4452ffd700 1 --2- 172.21.15.153:0/660840369 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f444c0692b0 0x7f444c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:10.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:10.881+0000 7f446259c700 1 --2- 172.21.15.153:0/660840369 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f444c0692b0 0x7f444c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:10.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:10.881+0000 7f4452ffd700 1 -- 172.21.15.153:0/660840369 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f4454088580 con 0x7f44640fef90 2024-03-11T23:00:10.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:10.881+0000 7f446259c700 1 --2- 172.21.15.153:0/660840369 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f444c0692b0 0x7f444c06b760 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4458006fd0 tx=0x7f4458006db0 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:00:10.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:10.885+0000 7f4452ffd700 1 -- 172.21.15.153:0/660840369 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f44540583e0 con 0x7f44640fef90 2024-03-11T23:00:11.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:11.114+0000 7f4468e86700 1 -- 172.21.15.153:0/660840369 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f446404bed0 con 0x7f44640fef90 2024-03-11T23:00:11.780 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:11 smithi163 ceph-mon[35584]: pgmap v180: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s wr, 7 op/s 2024-03-11T23:00:11.780 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:11 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/660840369' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-11T23:00:12.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:11 smithi153 ceph-mon[28962]: pgmap v180: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s wr, 7 op/s 2024-03-11T23:00:12.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:11 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/660840369' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-11T23:00:12.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:12.096+0000 7f4452ffd700 1 -- 172.21.15.153:0/660840369 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v12) v1 ==== 121+0+0 (secure 0 0 0) 0x7f4454057f70 con 0x7f44640fef90 2024-03-11T23:00:12.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:12.099+0000 7f4468e86700 1 -- 172.21.15.153:0/660840369 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f444c0692b0 msgr2=0x7f444c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:12.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:12.099+0000 7f4468e86700 1 --2- 172.21.15.153:0/660840369 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f444c0692b0 0x7f444c06b760 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4458006fd0 tx=0x7f4458006db0 comp rx=0 tx=0).stop 2024-03-11T23:00:12.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:12.099+0000 7f4468e86700 1 -- 172.21.15.153:0/660840369 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44640fef90 msgr2=0x7f446410e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:12.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:12.099+0000 7f4468e86700 1 --2- 172.21.15.153:0/660840369 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44640fef90 0x7f446410e270 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4454009fd0 tx=0x7f4454009e30 comp rx=0 tx=0).stop 2024-03-11T23:00:12.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:12.099+0000 7f4468e86700 1 -- 172.21.15.153:0/660840369 shutdown_connections 2024-03-11T23:00:12.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:12.099+0000 7f4468e86700 1 --2- 172.21.15.153:0/660840369 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f444c0692b0 0x7f444c06b760 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:12.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:12.099+0000 7f4468e86700 1 --2- 172.21.15.153:0/660840369 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44640fe640 0x7f446410dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:12.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:12.099+0000 7f4468e86700 1 --2- 172.21.15.153:0/660840369 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44640fef90 0x7f446410e270 secure :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4454009fd0 tx=0x7f4454009e30 comp rx=0 tx=0).stop 2024-03-11T23:00:12.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:12.099+0000 7f4468e86700 1 -- 172.21.15.153:0/660840369 >> 172.21.15.153:0/660840369 conn(0x7f44640f6d20 msgr2=0x7f4464101760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:00:12.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:12.099+0000 7f4468e86700 1 -- 172.21.15.153:0/660840369 shutdown_connections 2024-03-11T23:00:12.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:12.100+0000 7f4468e86700 1 -- 172.21.15.153:0/660840369 wait complete. 2024-03-11T23:00:12.774 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-11T23:00:12.784 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi153.front.sepia.ceph.com 2024-03-11T23:00:12.784 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2024-03-11T23:00:12.905 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:12 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:12.905 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:12 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:12.905 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:12 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:12.905 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:12 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:12.905 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:12 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:12.905 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:12 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:00:12.906 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:12 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/660840369' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-11T23:00:12.906 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:12 smithi163 ceph-mon[35584]: mds.? [v2:172.21.15.163:6826/1010460391,v1:172.21.15.163:6827/1010460391] up:boot 2024-03-11T23:00:12.906 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:12 smithi163 ceph-mon[35584]: pgmap v181: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s wr, 7 op/s 2024-03-11T23:00:12.906 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:12 smithi163 ceph-mon[35584]: fsmap cephfs:2 {0=cephfs.smithi153.qxjyxz=up:active,1=cephfs.smithi163.pmppga=up:active} 2 up:standby 2024-03-11T23:00:12.906 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:12 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi163.kzggfd"}]: dispatch 2024-03-11T23:00:12.906 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:12 smithi163 ceph-mon[35584]: fsmap cephfs:2 {0=cephfs.smithi153.qxjyxz=up:active,1=cephfs.smithi163.pmppga=up:active} 2 up:standby-replay 2024-03-11T23:00:13.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:12 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:13.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:12 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:13.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:12 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:13.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:12 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:13.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:12 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:13.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:12 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:00:13.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:12 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/660840369' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-11T23:00:13.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:12 smithi153 ceph-mon[28962]: mds.? [v2:172.21.15.163:6826/1010460391,v1:172.21.15.163:6827/1010460391] up:boot 2024-03-11T23:00:13.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:12 smithi153 ceph-mon[28962]: pgmap v181: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s wr, 7 op/s 2024-03-11T23:00:13.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:12 smithi153 ceph-mon[28962]: fsmap cephfs:2 {0=cephfs.smithi153.qxjyxz=up:active,1=cephfs.smithi163.pmppga=up:active} 2 up:standby 2024-03-11T23:00:13.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:12 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi163.kzggfd"}]: dispatch 2024-03-11T23:00:13.031 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:12 smithi153 ceph-mon[28962]: fsmap cephfs:2 {0=cephfs.smithi153.qxjyxz=up:active,1=cephfs.smithi163.pmppga=up:active} 2 up:standby-replay 2024-03-11T23:00:13.600 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T23:00:14.417 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:14 smithi153 ceph-mon[28962]: mds.? [v2:172.21.15.153:6826/633795058,v1:172.21.15.153:6827/633795058] up:active 2024-03-11T23:00:14.418 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:14 smithi153 ceph-mon[28962]: fsmap cephfs:2 {0=cephfs.smithi153.qxjyxz=up:active,1=cephfs.smithi163.pmppga=up:active} 2 up:standby-replay 2024-03-11T23:00:14.418 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:14 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:14.418 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:14 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:14.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:14 smithi163 ceph-mon[35584]: mds.? [v2:172.21.15.153:6826/633795058,v1:172.21.15.153:6827/633795058] up:active 2024-03-11T23:00:14.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:14 smithi163 ceph-mon[35584]: fsmap cephfs:2 {0=cephfs.smithi153.qxjyxz=up:active,1=cephfs.smithi163.pmppga=up:active} 2 up:standby-replay 2024-03-11T23:00:14.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:14 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:14.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:14 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:15.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:15.312+0000 7f05a7cee700 1 -- 172.21.15.153:0/2639154300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05a00fbe40 msgr2=0x7f05a00fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:15.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:15.312+0000 7f05a7cee700 1 --2- 172.21.15.153:0/2639154300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05a00fbe40 0x7f05a00fc250 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f059c004940 tx=0x7f059c009e30 comp rx=0 tx=0).stop 2024-03-11T23:00:15.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:15.312+0000 7f05a7cee700 1 -- 172.21.15.153:0/2639154300 shutdown_connections 2024-03-11T23:00:15.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:15.312+0000 7f05a7cee700 1 --2- 172.21.15.153:0/2639154300 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05a00fcfb0 0x7f05a00fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:15.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:15.312+0000 7f05a7cee700 1 --2- 172.21.15.153:0/2639154300 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05a00fbe40 0x7f05a00fc250 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:15.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:15.312+0000 7f05a7cee700 1 -- 172.21.15.153:0/2639154300 >> 172.21.15.153:0/2639154300 conn(0x7f05a00f7520 msgr2=0x7f05a00f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:00:15.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:15.312+0000 7f05a7cee700 1 -- 172.21.15.153:0/2639154300 shutdown_connections 2024-03-11T23:00:15.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:15.312+0000 7f05a7cee700 1 -- 172.21.15.153:0/2639154300 wait complete. 2024-03-11T23:00:15.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:15.312+0000 7f05a7cee700 1 Processor -- start 2024-03-11T23:00:15.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:15.312+0000 7f05a7cee700 1 -- start start 2024-03-11T23:00:15.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:15.313+0000 7f05a7cee700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05a00fbe40 0x7f05a010d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:15.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:15.313+0000 7f05a7cee700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05a00fcfb0 0x7f05a010de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:15.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:15.313+0000 7f05a7cee700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a010e470 con 0x7f05a00fcfb0 2024-03-11T23:00:15.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:15.313+0000 7f05a7cee700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a010ed10 con 0x7f05a00fbe40 2024-03-11T23:00:15.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:15.313+0000 7f05a5289700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05a00fcfb0 0x7f05a010de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:15.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:15.313+0000 7f05a5289700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05a00fcfb0 0x7f05a010de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55500/0 (socket says 172.21.15.153:55500) 2024-03-11T23:00:15.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:15.313+0000 7f05a5289700 1 -- 172.21.15.153:0/3522464710 learned_addr learned my addr 172.21.15.153:0/3522464710 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T23:00:15.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:15.313+0000 7f05a5a8a700 1 --2- 172.21.15.153:0/3522464710 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05a00fbe40 0x7f05a010d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:15.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:15.313+0000 7f05a5289700 1 -- 172.21.15.153:0/3522464710 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05a00fbe40 msgr2=0x7f05a010d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:15.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:15.313+0000 7f05a5289700 1 --2- 172.21.15.153:0/3522464710 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05a00fbe40 0x7f05a010d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:15.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:15.313+0000 7f05a5289700 1 -- 172.21.15.153:0/3522464710 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f059c005040 con 0x7f05a00fcfb0 2024-03-11T23:00:15.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:15.313+0000 7f05a5a8a700 1 --2- 172.21.15.153:0/3522464710 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05a00fbe40 0x7f05a010d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T23:00:15.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:15.313+0000 7f05a5289700 1 --2- 172.21.15.153:0/3522464710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05a00fcfb0 0x7f05a010de70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0590002a50 tx=0x7f05900046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:00:15.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:15.313+0000 7f05967fc700 1 -- 172.21.15.153:0/3522464710 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0590007cc0 con 0x7f05a00fcfb0 2024-03-11T23:00:15.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:15.313+0000 7f05967fc700 1 -- 172.21.15.153:0/3522464710 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f059000f040 con 0x7f05a00fcfb0 2024-03-11T23:00:15.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:15.313+0000 7f05a7cee700 1 -- 172.21.15.153:0/3522464710 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05a010eff0 con 0x7f05a00fcfb0 2024-03-11T23:00:15.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:15.313+0000 7f05967fc700 1 -- 172.21.15.153:0/3522464710 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0590009640 con 0x7f05a00fcfb0 2024-03-11T23:00:15.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:15.313+0000 7f05a7cee700 1 -- 172.21.15.153:0/3522464710 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05a010f530 con 0x7f05a00fcfb0 2024-03-11T23:00:15.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:15.314+0000 7f05a7cee700 1 -- 172.21.15.153:0/3522464710 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05a004bed0 con 0x7f05a00fcfb0 2024-03-11T23:00:15.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:15.315+0000 7f05967fc700 1 -- 172.21.15.153:0/3522464710 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f05900097a0 con 0x7f05a00fcfb0 2024-03-11T23:00:15.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:15.315+0000 7f05967fc700 1 --2- 172.21.15.153:0/3522464710 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f058c0692a0 0x7f058c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:15.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:15.315+0000 7f05967fc700 1 -- 172.21.15.153:0/3522464710 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0590086f00 con 0x7f05a00fcfb0 2024-03-11T23:00:15.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:15.315+0000 7f05a5a8a700 1 --2- 172.21.15.153:0/3522464710 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f058c0692a0 0x7f058c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:15.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:15.316+0000 7f05a5a8a700 1 --2- 172.21.15.153:0/3522464710 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f058c0692a0 0x7f058c06b750 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f059c004560 tx=0x7f059c009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:00:15.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:15.320+0000 7f05967fc700 1 -- 172.21.15.153:0/3522464710 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f0590056c70 con 0x7f05a00fcfb0 2024-03-11T23:00:15.324 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:15 smithi153 ceph-mon[28962]: pgmap v182: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s rd, 2.0 KiB/s wr, 8 op/s 2024-03-11T23:00:15.324 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:15 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:15.324 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:15 smithi153 ceph-mon[28962]: mds.? [v2:172.21.15.153:6828/2364288180,v1:172.21.15.153:6829/2364288180] up:standby-replay 2024-03-11T23:00:15.324 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:15 smithi153 ceph-mon[28962]: Dropping low affinity active daemon mds.cephfs.smithi163.pmppga in favor of higher affinity standby. 2024-03-11T23:00:15.324 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:15 smithi153 ceph-mon[28962]: Replacing daemon mds.cephfs.smithi163.pmppga as rank 1 with standby daemon mds.cephfs.smithi153.yuzdaj 2024-03-11T23:00:15.324 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:15 smithi153 ceph-mon[28962]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-11T23:00:15.324 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:15 smithi153 ceph-mon[28962]: fsmap cephfs:2 {0=cephfs.smithi153.qxjyxz=up:active,1=cephfs.smithi163.pmppga=up:active} 2 up:standby-replay 2024-03-11T23:00:15.324 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:15 smithi153 ceph-mon[28962]: osdmap e47: 6 total, 6 up, 6 in 2024-03-11T23:00:15.324 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:15 smithi153 ceph-mon[28962]: fsmap cephfs:2/2 {0=cephfs.smithi153.qxjyxz=up:active,1=cephfs.smithi153.yuzdaj=up:replay} 1 up:standby-replay 2024-03-11T23:00:15.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:15 smithi163 ceph-mon[35584]: pgmap v182: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s rd, 2.0 KiB/s wr, 8 op/s 2024-03-11T23:00:15.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:15 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:15.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:15 smithi163 ceph-mon[35584]: mds.? [v2:172.21.15.153:6828/2364288180,v1:172.21.15.153:6829/2364288180] up:standby-replay 2024-03-11T23:00:15.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:15 smithi163 ceph-mon[35584]: Dropping low affinity active daemon mds.cephfs.smithi163.pmppga in favor of higher affinity standby. 2024-03-11T23:00:15.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:15 smithi163 ceph-mon[35584]: Replacing daemon mds.cephfs.smithi163.pmppga as rank 1 with standby daemon mds.cephfs.smithi153.yuzdaj 2024-03-11T23:00:15.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:15 smithi163 ceph-mon[35584]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-11T23:00:15.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:15 smithi163 ceph-mon[35584]: fsmap cephfs:2 {0=cephfs.smithi153.qxjyxz=up:active,1=cephfs.smithi163.pmppga=up:active} 2 up:standby-replay 2024-03-11T23:00:15.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:15 smithi163 ceph-mon[35584]: osdmap e47: 6 total, 6 up, 6 in 2024-03-11T23:00:15.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:15 smithi163 ceph-mon[35584]: fsmap cephfs:2/2 {0=cephfs.smithi153.qxjyxz=up:active,1=cephfs.smithi153.yuzdaj=up:replay} 1 up:standby-replay 2024-03-11T23:00:15.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:15.562+0000 7f05a7cee700 1 -- 172.21.15.153:0/3522464710 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f05a0104930 con 0x7f05a00fcfb0 2024-03-11T23:00:16.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:16.149+0000 7f05967fc700 1 -- 172.21.15.153:0/3522464710 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v18) v1 ==== 168+0+0 (secure 0 0 0) 0x7f0590056800 con 0x7f05a00fcfb0 2024-03-11T23:00:16.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:16.152+0000 7f05a7cee700 1 -- 172.21.15.153:0/3522464710 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f058c0692a0 msgr2=0x7f058c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:16.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:16.152+0000 7f05a7cee700 1 --2- 172.21.15.153:0/3522464710 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f058c0692a0 0x7f058c06b750 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f059c004560 tx=0x7f059c009e30 comp rx=0 tx=0).stop 2024-03-11T23:00:16.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:16.152+0000 7f05a7cee700 1 -- 172.21.15.153:0/3522464710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05a00fcfb0 msgr2=0x7f05a010de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:16.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:16.152+0000 7f05a7cee700 1 --2- 172.21.15.153:0/3522464710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05a00fcfb0 0x7f05a010de70 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0590002a50 tx=0x7f05900046b0 comp rx=0 tx=0).stop 2024-03-11T23:00:16.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:16.152+0000 7f05a7cee700 1 -- 172.21.15.153:0/3522464710 shutdown_connections 2024-03-11T23:00:16.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:16.152+0000 7f05a7cee700 1 --2- 172.21.15.153:0/3522464710 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f058c0692a0 0x7f058c06b750 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:16.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:16.152+0000 7f05a7cee700 1 --2- 172.21.15.153:0/3522464710 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05a00fbe40 0x7f05a010d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:16.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:16.152+0000 7f05a7cee700 1 --2- 172.21.15.153:0/3522464710 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f05a00fcfb0 0x7f05a010de70 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:16.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:16.152+0000 7f05a7cee700 1 -- 172.21.15.153:0/3522464710 >> 172.21.15.153:0/3522464710 conn(0x7f05a00f7520 msgr2=0x7f05a0065f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:00:16.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:16.153+0000 7f05a7cee700 1 -- 172.21.15.153:0/3522464710 shutdown_connections 2024-03-11T23:00:16.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:16.153+0000 7f05a7cee700 1 -- 172.21.15.153:0/3522464710 wait complete. 2024-03-11T23:00:16.158 INFO:teuthology.orchestra.run.smithi153.stderr:inline data enabled 2024-03-11T23:00:16.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:16 smithi163 ceph-mon[35584]: mds.? [v2:172.21.15.163:6824/685283199,v1:172.21.15.163:6825/685283199] up:boot 2024-03-11T23:00:16.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:16 smithi163 ceph-mon[35584]: mds.? [v2:172.21.15.153:6828/2364288180,v1:172.21.15.153:6829/2364288180] up:resolve 2024-03-11T23:00:16.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:16 smithi163 ceph-mon[35584]: fsmap cephfs:2/2 {0=cephfs.smithi153.qxjyxz=up:active,1=cephfs.smithi153.yuzdaj=up:resolve} 1 up:standby-replay 1 up:standby 2024-03-11T23:00:16.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:16 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi163.pmppga"}]: dispatch 2024-03-11T23:00:16.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:16 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/3522464710' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T23:00:16.512 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:16 smithi153 ceph-mon[28962]: mds.? [v2:172.21.15.163:6824/685283199,v1:172.21.15.163:6825/685283199] up:boot 2024-03-11T23:00:16.512 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:16 smithi153 ceph-mon[28962]: mds.? [v2:172.21.15.153:6828/2364288180,v1:172.21.15.153:6829/2364288180] up:resolve 2024-03-11T23:00:16.512 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:16 smithi153 ceph-mon[28962]: fsmap cephfs:2/2 {0=cephfs.smithi153.qxjyxz=up:active,1=cephfs.smithi153.yuzdaj=up:resolve} 1 up:standby-replay 1 up:standby 2024-03-11T23:00:16.512 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:16 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi163.pmppga"}]: dispatch 2024-03-11T23:00:16.512 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:16 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/3522464710' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T23:00:16.819 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-11T23:00:16.830 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi153.front.sepia.ceph.com 2024-03-11T23:00:16.830 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-11T23:00:17.461 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T23:00:17.487 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:17 smithi163 ceph-mon[35584]: pgmap v184: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2024-03-11T23:00:17.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:17 smithi163 ceph-mon[35584]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-03-11T23:00:17.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:17 smithi163 ceph-mon[35584]: mds.? [v2:172.21.15.153:6828/2364288180,v1:172.21.15.153:6829/2364288180] up:reconnect 2024-03-11T23:00:17.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:17 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/3522464710' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T23:00:17.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:17 smithi163 ceph-mon[35584]: mds.? [v2:172.21.15.163:6826/1010460391,v1:172.21.15.163:6827/1010460391] up:standby-replay 2024-03-11T23:00:17.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:17 smithi163 ceph-mon[35584]: fsmap cephfs:2/2 {0=cephfs.smithi153.qxjyxz=up:active,1=cephfs.smithi153.yuzdaj=up:reconnect} 1 up:standby-replay 1 up:standby 2024-03-11T23:00:17.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:17 smithi153 ceph-mon[28962]: pgmap v184: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2024-03-11T23:00:17.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:17 smithi153 ceph-mon[28962]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-03-11T23:00:17.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:17 smithi153 ceph-mon[28962]: mds.? [v2:172.21.15.153:6828/2364288180,v1:172.21.15.153:6829/2364288180] up:reconnect 2024-03-11T23:00:17.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:17 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/3522464710' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T23:00:17.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:17 smithi153 ceph-mon[28962]: mds.? [v2:172.21.15.163:6826/1010460391,v1:172.21.15.163:6827/1010460391] up:standby-replay 2024-03-11T23:00:17.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:17 smithi153 ceph-mon[28962]: fsmap cephfs:2/2 {0=cephfs.smithi153.qxjyxz=up:active,1=cephfs.smithi153.yuzdaj=up:reconnect} 1 up:standby-replay 1 up:standby 2024-03-11T23:00:18.280 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:18 smithi153 ceph-mon[28962]: mds.? [v2:172.21.15.153:6828/2364288180,v1:172.21.15.153:6829/2364288180] up:rejoin 2024-03-11T23:00:18.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:18 smithi153 ceph-mon[28962]: fsmap cephfs:2/2 {0=cephfs.smithi153.qxjyxz=up:active,1=cephfs.smithi153.yuzdaj=up:rejoin} 1 up:standby-replay 1 up:standby 2024-03-11T23:00:18.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:18 smithi153 ceph-mon[28962]: daemon mds.cephfs.smithi153.yuzdaj is now active in filesystem cephfs as rank 1 2024-03-11T23:00:18.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:18 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:18.281 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:18 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:00:18.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:18 smithi163 ceph-mon[35584]: mds.? [v2:172.21.15.153:6828/2364288180,v1:172.21.15.153:6829/2364288180] up:rejoin 2024-03-11T23:00:18.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:18 smithi163 ceph-mon[35584]: fsmap cephfs:2/2 {0=cephfs.smithi153.qxjyxz=up:active,1=cephfs.smithi153.yuzdaj=up:rejoin} 1 up:standby-replay 1 up:standby 2024-03-11T23:00:18.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:18 smithi163 ceph-mon[35584]: daemon mds.cephfs.smithi153.yuzdaj is now active in filesystem cephfs as rank 1 2024-03-11T23:00:18.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:18 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:18.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:18 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:00:19.388 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:19 smithi153 ceph-mon[28962]: pgmap v185: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 1.6 KiB/s wr, 8 op/s 2024-03-11T23:00:19.388 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:19 smithi153 ceph-mon[28962]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-11T23:00:19.389 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:19 smithi153 ceph-mon[28962]: mds.? [v2:172.21.15.153:6828/2364288180,v1:172.21.15.153:6829/2364288180] up:active 2024-03-11T23:00:19.389 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:19 smithi153 ceph-mon[28962]: fsmap cephfs:2 {0=cephfs.smithi153.qxjyxz=up:active,1=cephfs.smithi153.yuzdaj=up:active} 1 up:standby-replay 1 up:standby 2024-03-11T23:00:19.389 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:19 smithi153 ceph-mon[28962]: fsmap cephfs:2 {0=cephfs.smithi153.qxjyxz=up:active,1=cephfs.smithi153.yuzdaj=up:active} 2 up:standby-replay 2024-03-11T23:00:19.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:19 smithi163 ceph-mon[35584]: pgmap v185: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 1.6 KiB/s wr, 8 op/s 2024-03-11T23:00:19.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:19 smithi163 ceph-mon[35584]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-11T23:00:19.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:19 smithi163 ceph-mon[35584]: mds.? [v2:172.21.15.153:6828/2364288180,v1:172.21.15.153:6829/2364288180] up:active 2024-03-11T23:00:19.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:19 smithi163 ceph-mon[35584]: fsmap cephfs:2 {0=cephfs.smithi153.qxjyxz=up:active,1=cephfs.smithi153.yuzdaj=up:active} 1 up:standby-replay 1 up:standby 2024-03-11T23:00:19.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:19 smithi163 ceph-mon[35584]: fsmap cephfs:2 {0=cephfs.smithi153.qxjyxz=up:active,1=cephfs.smithi153.yuzdaj=up:active} 2 up:standby-replay 2024-03-11T23:00:20.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.226+0000 7f2a38988700 1 -- 172.21.15.153:0/2063433988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a340fb220 msgr2=0x7f2a340fb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:20.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.226+0000 7f2a38988700 1 --2- 172.21.15.153:0/2063433988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a340fb220 0x7f2a340fb630 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2a24004940 tx=0x7f2a24009e30 comp rx=0 tx=0).stop 2024-03-11T23:00:20.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.226+0000 7f2a38988700 1 -- 172.21.15.153:0/2063433988 shutdown_connections 2024-03-11T23:00:20.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.226+0000 7f2a38988700 1 --2- 172.21.15.153:0/2063433988 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2a340fe2b0 0x7f2a340fe700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:20.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.226+0000 7f2a38988700 1 --2- 172.21.15.153:0/2063433988 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a340fb220 0x7f2a340fb630 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:20.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.226+0000 7f2a38988700 1 -- 172.21.15.153:0/2063433988 >> 172.21.15.153:0/2063433988 conn(0x7f2a340f6ca0 msgr2=0x7f2a340f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:00:20.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.226+0000 7f2a38988700 1 -- 172.21.15.153:0/2063433988 shutdown_connections 2024-03-11T23:00:20.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.226+0000 7f2a38988700 1 -- 172.21.15.153:0/2063433988 wait complete. 2024-03-11T23:00:20.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.226+0000 7f2a38988700 1 Processor -- start 2024-03-11T23:00:20.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.227+0000 7f2a38988700 1 -- start start 2024-03-11T23:00:20.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.227+0000 7f2a38988700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a340fb220 0x7f2a3410da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:20.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.227+0000 7f2a38988700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2a340fe2b0 0x7f2a3410df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:20.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.227+0000 7f2a38988700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3410e580 con 0x7f2a340fb220 2024-03-11T23:00:20.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.227+0000 7f2a38988700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3410ee20 con 0x7f2a340fe2b0 2024-03-11T23:00:20.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.227+0000 7f2a3259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a340fb220 0x7f2a3410da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:20.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.227+0000 7f2a3259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a340fb220 0x7f2a3410da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55514/0 (socket says 172.21.15.153:55514) 2024-03-11T23:00:20.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.227+0000 7f2a3259c700 1 -- 172.21.15.153:0/3907801085 learned_addr learned my addr 172.21.15.153:0/3907801085 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T23:00:20.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.227+0000 7f2a29d9b700 1 --2- 172.21.15.153:0/3907801085 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2a340fe2b0 0x7f2a3410df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:20.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.227+0000 7f2a3259c700 1 -- 172.21.15.153:0/3907801085 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2a340fe2b0 msgr2=0x7f2a3410df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:20.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.227+0000 7f2a3259c700 1 --2- 172.21.15.153:0/3907801085 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2a340fe2b0 0x7f2a3410df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:20.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.227+0000 7f2a3259c700 1 -- 172.21.15.153:0/3907801085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a24005040 con 0x7f2a340fb220 2024-03-11T23:00:20.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.227+0000 7f2a3259c700 1 --2- 172.21.15.153:0/3907801085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a340fb220 0x7f2a3410da40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2a24009150 tx=0x7f2a24009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:00:20.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.227+0000 7f2a2b7fe700 1 -- 172.21.15.153:0/3907801085 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a2400e040 con 0x7f2a340fb220 2024-03-11T23:00:20.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.227+0000 7f2a2b7fe700 1 -- 172.21.15.153:0/3907801085 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2a2400bbf0 con 0x7f2a340fb220 2024-03-11T23:00:20.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.227+0000 7f2a38988700 1 -- 172.21.15.153:0/3907801085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a3410f0a0 con 0x7f2a340fb220 2024-03-11T23:00:20.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.227+0000 7f2a2b7fe700 1 -- 172.21.15.153:0/3907801085 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a2401c3f0 con 0x7f2a340fb220 2024-03-11T23:00:20.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.227+0000 7f2a38988700 1 -- 172.21.15.153:0/3907801085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a3410f5e0 con 0x7f2a340fb220 2024-03-11T23:00:20.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.228+0000 7f2a38988700 1 -- 172.21.15.153:0/3907801085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a3404bed0 con 0x7f2a340fb220 2024-03-11T23:00:20.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.229+0000 7f2a2b7fe700 1 -- 172.21.15.153:0/3907801085 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f2a2400bd60 con 0x7f2a340fb220 2024-03-11T23:00:20.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.229+0000 7f2a2b7fe700 1 --2- 172.21.15.153:0/3907801085 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f2a200692a0 0x7f2a2006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:20.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.229+0000 7f2a2b7fe700 1 -- 172.21.15.153:0/3907801085 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f2a24088500 con 0x7f2a340fb220 2024-03-11T23:00:20.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.229+0000 7f2a29d9b700 1 --2- 172.21.15.153:0/3907801085 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f2a200692a0 0x7f2a2006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:20.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.230+0000 7f2a29d9b700 1 --2- 172.21.15.153:0/3907801085 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f2a200692a0 0x7f2a2006b750 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c007fd0 tx=0x7f2a1c009040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:00:20.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.234+0000 7f2a2b7fe700 1 -- 172.21.15.153:0/3907801085 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f2a24058310 con 0x7f2a340fb220 2024-03-11T23:00:20.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.516+0000 7f2a38988700 1 -- 172.21.15.153:0/3907801085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f2a3410f9b0 con 0x7f2a340fb220 2024-03-11T23:00:20.521 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.517+0000 7f2a2b7fe700 1 -- 172.21.15.153:0/3907801085 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 21 v21) v1 ==== 76+0+1833 (secure 0 0 0) 0x7f2a24057ea0 con 0x7f2a340fb220 2024-03-11T23:00:20.521 INFO:teuthology.orchestra.run.smithi153.stdout:e21 2024-03-11T23:00:20.521 INFO:teuthology.orchestra.run.smithi153.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T23:00:20.521 INFO:teuthology.orchestra.run.smithi153.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-11T23:00:20.522 INFO:teuthology.orchestra.run.smithi153.stdout:legacy client fscid: 1 2024-03-11T23:00:20.522 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T23:00:20.522 INFO:teuthology.orchestra.run.smithi153.stdout:Filesystem 'cephfs' (1) 2024-03-11T23:00:20.522 INFO:teuthology.orchestra.run.smithi153.stdout:fs_name cephfs 2024-03-11T23:00:20.522 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 21 2024-03-11T23:00:20.522 INFO:teuthology.orchestra.run.smithi153.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-11T23:00:20.522 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-03-11T22:59:59.460410+0000 2024-03-11T23:00:20.522 INFO:teuthology.orchestra.run.smithi153.stdout:modified 2024-03-11T23:00:18.156048+0000 2024-03-11T23:00:20.522 INFO:teuthology.orchestra.run.smithi153.stdout:tableserver 0 2024-03-11T23:00:20.522 INFO:teuthology.orchestra.run.smithi153.stdout:root 0 2024-03-11T23:00:20.522 INFO:teuthology.orchestra.run.smithi153.stdout:session_timeout 60 2024-03-11T23:00:20.522 INFO:teuthology.orchestra.run.smithi153.stdout:session_autoclose 300 2024-03-11T23:00:20.522 INFO:teuthology.orchestra.run.smithi153.stdout:max_file_size 1099511627776 2024-03-11T23:00:20.523 INFO:teuthology.orchestra.run.smithi153.stdout:required_client_features {} 2024-03-11T23:00:20.525 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure 0 2024-03-11T23:00:20.525 INFO:teuthology.orchestra.run.smithi153.stdout:last_failure_osd_epoch 47 2024-03-11T23:00:20.525 INFO:teuthology.orchestra.run.smithi153.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-11T23:00:20.525 INFO:teuthology.orchestra.run.smithi153.stdout:max_mds 2 2024-03-11T23:00:20.525 INFO:teuthology.orchestra.run.smithi153.stdout:in 0,1 2024-03-11T23:00:20.526 INFO:teuthology.orchestra.run.smithi153.stdout:up {0=14444,1=14460} 2024-03-11T23:00:20.526 INFO:teuthology.orchestra.run.smithi153.stdout:failed 2024-03-11T23:00:20.526 INFO:teuthology.orchestra.run.smithi153.stdout:damaged 2024-03-11T23:00:20.526 INFO:teuthology.orchestra.run.smithi153.stdout:stopped 2024-03-11T23:00:20.526 INFO:teuthology.orchestra.run.smithi153.stdout:data_pools [3] 2024-03-11T23:00:20.526 INFO:teuthology.orchestra.run.smithi153.stdout:metadata_pool 2 2024-03-11T23:00:20.526 INFO:teuthology.orchestra.run.smithi153.stdout:inline_data enabled 2024-03-11T23:00:20.527 INFO:teuthology.orchestra.run.smithi153.stdout:balancer 2024-03-11T23:00:20.527 INFO:teuthology.orchestra.run.smithi153.stdout:standby_count_wanted 1 2024-03-11T23:00:20.527 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.qxjyxz{0:14444} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.153:6826/633795058,v1:172.21.15.153:6827/633795058] compat {c=[1],r=[1],i=[7ff]}] 2024-03-11T23:00:20.527 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi163.kzggfd{0:24263} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.163:6826/1010460391,v1:172.21.15.163:6827/1010460391] compat {c=[1],r=[1],i=[7ff]}] 2024-03-11T23:00:20.527 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi153.yuzdaj{1:14460} state up:active seq 6 join_fscid=1 addr [v2:172.21.15.153:6828/2364288180,v1:172.21.15.153:6829/2364288180] compat {c=[1],r=[1],i=[7ff]}] 2024-03-11T23:00:20.527 INFO:teuthology.orchestra.run.smithi153.stdout:[mds.cephfs.smithi163.pmppga{1:24267} state up:standby-replay seq 1 join_fscid=1 addr [v2:172.21.15.163:6824/685283199,v1:172.21.15.163:6825/685283199] compat {c=[1],r=[1],i=[7ff]}] 2024-03-11T23:00:20.527 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T23:00:20.527 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T23:00:20.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.519+0000 7f2a38988700 1 -- 172.21.15.153:0/3907801085 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f2a200692a0 msgr2=0x7f2a2006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:20.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.519+0000 7f2a38988700 1 --2- 172.21.15.153:0/3907801085 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f2a200692a0 0x7f2a2006b750 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c007fd0 tx=0x7f2a1c009040 comp rx=0 tx=0).stop 2024-03-11T23:00:20.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.519+0000 7f2a38988700 1 -- 172.21.15.153:0/3907801085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a340fb220 msgr2=0x7f2a3410da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:20.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.519+0000 7f2a38988700 1 --2- 172.21.15.153:0/3907801085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a340fb220 0x7f2a3410da40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2a24009150 tx=0x7f2a24009e30 comp rx=0 tx=0).stop 2024-03-11T23:00:20.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.519+0000 7f2a38988700 1 -- 172.21.15.153:0/3907801085 shutdown_connections 2024-03-11T23:00:20.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.519+0000 7f2a38988700 1 --2- 172.21.15.153:0/3907801085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a340fb220 0x7f2a3410da40 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:20.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.519+0000 7f2a38988700 1 --2- 172.21.15.153:0/3907801085 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f2a200692a0 0x7f2a2006b750 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:20.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.519+0000 7f2a38988700 1 --2- 172.21.15.153:0/3907801085 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2a340fe2b0 0x7f2a3410df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:20.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.519+0000 7f2a38988700 1 -- 172.21.15.153:0/3907801085 >> 172.21.15.153:0/3907801085 conn(0x7f2a340f6ca0 msgr2=0x7f2a341014e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:00:20.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.519+0000 7f2a38988700 1 -- 172.21.15.153:0/3907801085 shutdown_connections 2024-03-11T23:00:20.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:20.519+0000 7f2a38988700 1 -- 172.21.15.153:0/3907801085 wait complete. 2024-03-11T23:00:20.529 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 21 2024-03-11T23:00:21.268 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-03-11T23:00:21.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:21 smithi153 ceph-mon[28962]: pgmap v186: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.4 KiB/s rd, 614 B/s wr, 8 op/s 2024-03-11T23:00:21.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:21 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:21.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:21 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:21.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:21 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:00:21.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:21 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/3907801085' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-11T23:00:21.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:21 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:00:21.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:21 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:21.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:21 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:00:21.878 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:21 smithi163 ceph-mon[35584]: pgmap v186: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.4 KiB/s rd, 614 B/s wr, 8 op/s 2024-03-11T23:00:21.878 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:21 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:21.878 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:21 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:21.878 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:21 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:00:21.878 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:21 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/3907801085' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-11T23:00:21.879 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:21 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:00:21.879 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:21 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:21.879 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:21 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:00:22.177 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T23:00:23.923 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:23 smithi153 ceph-mon[28962]: pgmap v187: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 7.4 KiB/s rd, 0 B/s wr, 8 op/s 2024-03-11T23:00:23.924 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:23 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:23.924 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:23 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:23.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:23 smithi163 ceph-mon[35584]: pgmap v187: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 7.4 KiB/s rd, 0 B/s wr, 8 op/s 2024-03-11T23:00:23.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:23 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:23.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:23 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:25.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.073+0000 7f8fa0bbb700 1 -- 172.21.15.153:0/2724309455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f9c0fc7f0 msgr2=0x7f8f9c0fcc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:25.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.073+0000 7f8fa0bbb700 1 --2- 172.21.15.153:0/2724309455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f9c0fc7f0 0x7f8f9c0fcc40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8f90004a50 tx=0x7f8f9000af40 comp rx=0 tx=0).stop 2024-03-11T23:00:25.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.073+0000 7f8fa0bbb700 1 -- 172.21.15.153:0/2724309455 shutdown_connections 2024-03-11T23:00:25.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.073+0000 7f8fa0bbb700 1 --2- 172.21.15.153:0/2724309455 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f9c0fc7f0 0x7f8f9c0fcc40 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:25.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.073+0000 7f8fa0bbb700 1 --2- 172.21.15.153:0/2724309455 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f9c0fb5e0 0x7f8f9c0fb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:25.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.073+0000 7f8fa0bbb700 1 -- 172.21.15.153:0/2724309455 >> 172.21.15.153:0/2724309455 conn(0x7f8f9c0f6ca0 msgr2=0x7f8f9c0f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:00:25.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.073+0000 7f8fa0bbb700 1 -- 172.21.15.153:0/2724309455 shutdown_connections 2024-03-11T23:00:25.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.073+0000 7f8fa0bbb700 1 -- 172.21.15.153:0/2724309455 wait complete. 2024-03-11T23:00:25.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.074+0000 7f8fa0bbb700 1 Processor -- start 2024-03-11T23:00:25.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.074+0000 7f8fa0bbb700 1 -- start start 2024-03-11T23:00:25.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.074+0000 7f8fa0bbb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f9c0fb5e0 0x7f8f9c10dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:25.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.074+0000 7f8fa0bbb700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f9c0fc7f0 0x7f8f9c10dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:25.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.074+0000 7f8fa0bbb700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9c10e680 con 0x7f8f9c0fb5e0 2024-03-11T23:00:25.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.074+0000 7f8fa0bbb700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9c10eec0 con 0x7f8f9c0fc7f0 2024-03-11T23:00:25.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.074+0000 7f8f9a59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f9c0fb5e0 0x7f8f9c10dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:25.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.074+0000 7f8f9a59c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f9c0fb5e0 0x7f8f9c10dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:58534/0 (socket says 172.21.15.153:58534) 2024-03-11T23:00:25.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.074+0000 7f8f9a59c700 1 -- 172.21.15.153:0/2561071468 learned_addr learned my addr 172.21.15.153:0/2561071468 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T23:00:25.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.074+0000 7f8f99d9b700 1 --2- 172.21.15.153:0/2561071468 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f9c0fc7f0 0x7f8f9c10dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:25.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.074+0000 7f8f9a59c700 1 -- 172.21.15.153:0/2561071468 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f9c0fc7f0 msgr2=0x7f8f9c10dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:25.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.074+0000 7f8f9a59c700 1 --2- 172.21.15.153:0/2561071468 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f9c0fc7f0 0x7f8f9c10dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:25.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.074+0000 7f8f9a59c700 1 -- 172.21.15.153:0/2561071468 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f90005040 con 0x7f8f9c0fb5e0 2024-03-11T23:00:25.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.074+0000 7f8f99d9b700 1 --2- 172.21.15.153:0/2561071468 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f9c0fc7f0 0x7f8f9c10dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-11T23:00:25.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.074+0000 7f8f9a59c700 1 --2- 172.21.15.153:0/2561071468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f9c0fb5e0 0x7f8f9c10dab0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c00b790 tx=0x7f8f8c009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:00:25.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.074+0000 7f8f8affd700 1 -- 172.21.15.153:0/2561071468 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f8c00f6f0 con 0x7f8f9c0fb5e0 2024-03-11T23:00:25.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.074+0000 7f8fa0bbb700 1 -- 172.21.15.153:0/2561071468 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f9c10f1a0 con 0x7f8f9c0fb5e0 2024-03-11T23:00:25.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.074+0000 7f8f8affd700 1 -- 172.21.15.153:0/2561071468 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8f8c00fd20 con 0x7f8f9c0fb5e0 2024-03-11T23:00:25.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.074+0000 7f8f8affd700 1 -- 172.21.15.153:0/2561071468 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f8c0103a0 con 0x7f8f9c0fb5e0 2024-03-11T23:00:25.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.074+0000 7f8fa0bbb700 1 -- 172.21.15.153:0/2561071468 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f9c10f6e0 con 0x7f8f9c0fb5e0 2024-03-11T23:00:25.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.075+0000 7f8fa0bbb700 1 -- 172.21.15.153:0/2561071468 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f9c107930 con 0x7f8f9c0fb5e0 2024-03-11T23:00:25.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.076+0000 7f8f8affd700 1 -- 172.21.15.153:0/2561071468 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f8f8c011400 con 0x7f8f9c0fb5e0 2024-03-11T23:00:25.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.077+0000 7f8f8affd700 1 --2- 172.21.15.153:0/2561071468 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f8f84069300 0x7f8f8406b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:25.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.077+0000 7f8f8affd700 1 -- 172.21.15.153:0/2561071468 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f8f8c087120 con 0x7f8f9c0fb5e0 2024-03-11T23:00:25.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.077+0000 7f8f99d9b700 1 --2- 172.21.15.153:0/2561071468 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f8f84069300 0x7f8f8406b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:25.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.077+0000 7f8f99d9b700 1 --2- 172.21.15.153:0/2561071468 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f8f84069300 0x7f8f8406b7b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8f90008d30 tx=0x7f8f9000af40 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:00:25.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.081+0000 7f8f8affd700 1 -- 172.21.15.153:0/2561071468 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f8f8c056d70 con 0x7f8f9c0fb5e0 2024-03-11T23:00:25.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.313+0000 7f8fa0bbb700 1 -- 172.21.15.153:0/2561071468 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f9c04bed0 con 0x7f8f9c0fb5e0 2024-03-11T23:00:25.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.313+0000 7f8f8affd700 1 -- 172.21.15.153:0/2561071468 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+4725 (secure 0 0 0) 0x7f8f8c056900 con 0x7f8f9c0fb5e0 2024-03-11T23:00:25.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.315+0000 7f8fa0bbb700 1 -- 172.21.15.153:0/2561071468 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f8f84069300 msgr2=0x7f8f8406b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:25.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.315+0000 7f8fa0bbb700 1 --2- 172.21.15.153:0/2561071468 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f8f84069300 0x7f8f8406b7b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8f90008d30 tx=0x7f8f9000af40 comp rx=0 tx=0).stop 2024-03-11T23:00:25.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.315+0000 7f8fa0bbb700 1 -- 172.21.15.153:0/2561071468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f9c0fb5e0 msgr2=0x7f8f9c10dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:25.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.315+0000 7f8fa0bbb700 1 --2- 172.21.15.153:0/2561071468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f9c0fb5e0 0x7f8f9c10dab0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c00b790 tx=0x7f8f8c009220 comp rx=0 tx=0).stop 2024-03-11T23:00:25.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.316+0000 7f8fa0bbb700 1 -- 172.21.15.153:0/2561071468 shutdown_connections 2024-03-11T23:00:25.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.316+0000 7f8fa0bbb700 1 --2- 172.21.15.153:0/2561071468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8f9c0fb5e0 0x7f8f9c10dab0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:25.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.316+0000 7f8fa0bbb700 1 --2- 172.21.15.153:0/2561071468 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f8f84069300 0x7f8f8406b7b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:25.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.316+0000 7f8fa0bbb700 1 --2- 172.21.15.153:0/2561071468 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f9c0fc7f0 0x7f8f9c10dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:25.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.316+0000 7f8fa0bbb700 1 -- 172.21.15.153:0/2561071468 >> 172.21.15.153:0/2561071468 conn(0x7f8f9c0f6ca0 msgr2=0x7f8f9c0ffc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:00:25.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.316+0000 7f8fa0bbb700 1 -- 172.21.15.153:0/2561071468 shutdown_connections 2024-03-11T23:00:25.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:25.316+0000 7f8fa0bbb700 1 -- 172.21.15.153:0/2561071468 wait complete. 2024-03-11T23:00:25.321 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 21 2024-03-11T23:00:25.331 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-03-11T23:00:25.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:25 smithi163 ceph-mon[35584]: pgmap v188: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 8 op/s 2024-03-11T23:00:25.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:25 smithi153 ceph-mon[28962]: pgmap v188: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 6.8 KiB/s rd, 0 B/s wr, 8 op/s 2024-03-11T23:00:26.478 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-03-11T23:00:26.655 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:26 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/2561071468' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T23:00:26.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:26 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/2561071468' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T23:00:27.080 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T23:00:27.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:27 smithi153 ceph-mon[28962]: pgmap v189: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.7 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-11T23:00:27.639 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:27 smithi163 ceph-mon[35584]: pgmap v189: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.7 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-11T23:00:28.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:28.730+0000 7fcf6c27b700 1 -- 172.21.15.153:0/3456819229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf640fcef0 msgr2=0x7fcf640fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:28.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:28.730+0000 7fcf6c27b700 1 --2- 172.21.15.153:0/3456819229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf640fcef0 0x7fcf640fd320 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fcf60004940 tx=0x7fcf60009e30 comp rx=0 tx=0).stop 2024-03-11T23:00:28.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:28.730+0000 7fcf6c27b700 1 -- 172.21.15.153:0/3456819229 shutdown_connections 2024-03-11T23:00:28.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:28.730+0000 7fcf6c27b700 1 --2- 172.21.15.153:0/3456819229 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcf640fe0e0 0x7fcf640fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:28.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:28.731+0000 7fcf6c27b700 1 --2- 172.21.15.153:0/3456819229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf640fcef0 0x7fcf640fd320 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:28.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:28.731+0000 7fcf6c27b700 1 -- 172.21.15.153:0/3456819229 >> 172.21.15.153:0/3456819229 conn(0x7fcf640f8620 msgr2=0x7fcf640faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:00:28.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:28.731+0000 7fcf6c27b700 1 -- 172.21.15.153:0/3456819229 shutdown_connections 2024-03-11T23:00:28.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:28.731+0000 7fcf6c27b700 1 -- 172.21.15.153:0/3456819229 wait complete. 2024-03-11T23:00:28.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:28.731+0000 7fcf6c27b700 1 Processor -- start 2024-03-11T23:00:28.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:28.731+0000 7fcf6c27b700 1 -- start start 2024-03-11T23:00:28.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:28.731+0000 7fcf6c27b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcf640fcef0 0x7fcf6410d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:28.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:28.731+0000 7fcf6c27b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf640fe0e0 0x7fcf6410de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:28.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:28.731+0000 7fcf6c27b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6410e430 con 0x7fcf640fe0e0 2024-03-11T23:00:28.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:28.731+0000 7fcf6c27b700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6410ecd0 con 0x7fcf640fcef0 2024-03-11T23:00:28.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:28.732+0000 7fcf69816700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf640fe0e0 0x7fcf6410de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:28.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:28.732+0000 7fcf69816700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf640fe0e0 0x7fcf6410de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:58550/0 (socket says 172.21.15.153:58550) 2024-03-11T23:00:28.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:28.732+0000 7fcf69816700 1 -- 172.21.15.153:0/4266776863 learned_addr learned my addr 172.21.15.153:0/4266776863 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T23:00:28.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:28.732+0000 7fcf6a017700 1 --2- 172.21.15.153:0/4266776863 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcf640fcef0 0x7fcf6410d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:28.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:28.732+0000 7fcf69816700 1 -- 172.21.15.153:0/4266776863 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcf640fcef0 msgr2=0x7fcf6410d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:28.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:28.732+0000 7fcf69816700 1 --2- 172.21.15.153:0/4266776863 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcf640fcef0 0x7fcf6410d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:28.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:28.732+0000 7fcf69816700 1 -- 172.21.15.153:0/4266776863 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf60005040 con 0x7fcf640fe0e0 2024-03-11T23:00:28.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:28.732+0000 7fcf69816700 1 --2- 172.21.15.153:0/4266776863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf640fe0e0 0x7fcf6410de30 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fcf5400eb40 tx=0x7fcf5400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:00:28.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:28.732+0000 7fcf5affd700 1 -- 172.21.15.153:0/4266776863 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf54007cc0 con 0x7fcf640fe0e0 2024-03-11T23:00:28.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:28.732+0000 7fcf5affd700 1 -- 172.21.15.153:0/4266776863 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcf5400f040 con 0x7fcf640fe0e0 2024-03-11T23:00:28.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:28.732+0000 7fcf6c27b700 1 -- 172.21.15.153:0/4266776863 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf6410ef50 con 0x7fcf640fe0e0 2024-03-11T23:00:28.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:28.732+0000 7fcf5affd700 1 -- 172.21.15.153:0/4266776863 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf5400ca10 con 0x7fcf640fe0e0 2024-03-11T23:00:28.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:28.732+0000 7fcf6c27b700 1 -- 172.21.15.153:0/4266776863 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf6410f490 con 0x7fcf640fe0e0 2024-03-11T23:00:28.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:28.733+0000 7fcf6c27b700 1 -- 172.21.15.153:0/4266776863 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf6404bed0 con 0x7fcf640fe0e0 2024-03-11T23:00:28.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:28.734+0000 7fcf5affd700 1 -- 172.21.15.153:0/4266776863 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fcf5400cc80 con 0x7fcf640fe0e0 2024-03-11T23:00:28.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:28.734+0000 7fcf5affd700 1 --2- 172.21.15.153:0/4266776863 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fcf500692a0 0x7fcf5006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:28.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:28.734+0000 7fcf5affd700 1 -- 172.21.15.153:0/4266776863 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fcf54086dd0 con 0x7fcf640fe0e0 2024-03-11T23:00:28.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:28.734+0000 7fcf6a017700 1 --2- 172.21.15.153:0/4266776863 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fcf500692a0 0x7fcf5006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:28.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:28.735+0000 7fcf6a017700 1 --2- 172.21.15.153:0/4266776863 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fcf500692a0 0x7fcf5006b750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fcf60000c00 tx=0x7fcf60009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:00:28.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:28.739+0000 7fcf5affd700 1 -- 172.21.15.153:0/4266776863 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7fcf5404dd20 con 0x7fcf640fe0e0 2024-03-11T23:00:29.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.007+0000 7fcf6c27b700 1 -- 172.21.15.153:0/4266776863 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7fcf64105a30 con 0x7fcf640fe0e0 2024-03-11T23:00:29.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.007+0000 7fcf5affd700 1 -- 172.21.15.153:0/4266776863 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v21) v1 ==== 78+0+99 (secure 0 0 0) 0x7fcf54056b70 con 0x7fcf640fe0e0 2024-03-11T23:00:29.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.009+0000 7fcf6c27b700 1 -- 172.21.15.153:0/4266776863 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fcf500692a0 msgr2=0x7fcf5006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:29.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.009+0000 7fcf6c27b700 1 --2- 172.21.15.153:0/4266776863 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fcf500692a0 0x7fcf5006b750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fcf60000c00 tx=0x7fcf60009e30 comp rx=0 tx=0).stop 2024-03-11T23:00:29.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.009+0000 7fcf6c27b700 1 -- 172.21.15.153:0/4266776863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf640fe0e0 msgr2=0x7fcf6410de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:29.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.009+0000 7fcf6c27b700 1 --2- 172.21.15.153:0/4266776863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf640fe0e0 0x7fcf6410de30 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fcf5400eb40 tx=0x7fcf5400b040 comp rx=0 tx=0).stop 2024-03-11T23:00:29.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.009+0000 7fcf6c27b700 1 -- 172.21.15.153:0/4266776863 shutdown_connections 2024-03-11T23:00:29.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.009+0000 7fcf6c27b700 1 --2- 172.21.15.153:0/4266776863 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fcf500692a0 0x7fcf5006b750 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:29.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.009+0000 7fcf6c27b700 1 --2- 172.21.15.153:0/4266776863 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcf640fcef0 0x7fcf6410d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:29.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.009+0000 7fcf6c27b700 1 --2- 172.21.15.153:0/4266776863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcf640fe0e0 0x7fcf6410de30 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:29.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.009+0000 7fcf6c27b700 1 -- 172.21.15.153:0/4266776863 >> 172.21.15.153:0/4266776863 conn(0x7fcf640f8620 msgr2=0x7fcf641013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:00:29.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.009+0000 7fcf6c27b700 1 -- 172.21.15.153:0/4266776863 shutdown_connections 2024-03-11T23:00:29.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.009+0000 7fcf6c27b700 1 -- 172.21.15.153:0/4266776863 wait complete. 2024-03-11T23:00:29.024 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-03-11T23:00:29.570 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:29 smithi153 ceph-mon[28962]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-11T23:00:29.570 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:29 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:29.570 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:29 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:29.570 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:29 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:00:29.571 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:29 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:00:29.571 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:29 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:29.571 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:29 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/4266776863' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-03-11T23:00:29.595 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-03-11T23:00:29.608 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T23:00:29.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.723+0000 7f928ed1d640 1 -- 172.21.15.153:0/516954362 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92880fb180 msgr2=0x7f92880fb590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:29.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.723+0000 7f928ed1d640 1 --2- 172.21.15.153:0/516954362 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92880fb180 0x7f92880fb590 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f927c0048d0 tx=0x7f927c009da0 comp rx=0 tx=0).stop 2024-03-11T23:00:29.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.724+0000 7f928ed1d640 1 -- 172.21.15.153:0/516954362 shutdown_connections 2024-03-11T23:00:29.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.724+0000 7f928ed1d640 1 --2- 172.21.15.153:0/516954362 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92880fc450 0x7f92880fc8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:29.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.724+0000 7f928ed1d640 1 --2- 172.21.15.153:0/516954362 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92880fb180 0x7f92880fb590 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:29.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.724+0000 7f928ed1d640 1 -- 172.21.15.153:0/516954362 >> 172.21.15.153:0/516954362 conn(0x7f92880f6940 msgr2=0x7f92880f8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:00:29.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.724+0000 7f928ed1d640 1 -- 172.21.15.153:0/516954362 shutdown_connections 2024-03-11T23:00:29.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.724+0000 7f928ed1d640 1 -- 172.21.15.153:0/516954362 wait complete. 2024-03-11T23:00:29.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.724+0000 7f928ed1d640 1 Processor -- start 2024-03-11T23:00:29.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.724+0000 7f928ed1d640 1 -- start start 2024-03-11T23:00:29.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.724+0000 7f928ed1d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92880fb180 0x7f928810cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:29.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.725+0000 7f928ed1d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92880fc450 0x7f928810d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:29.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.725+0000 7f928ed1d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f928810d6f0 con 0x7f92880fc450 2024-03-11T23:00:29.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.725+0000 7f928ed1d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9288112140 con 0x7f92880fb180 2024-03-11T23:00:29.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.725+0000 7f9287fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92880fc450 0x7f928810d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:29.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.725+0000 7f9287fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92880fc450 0x7f928810d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:58570/0 (socket says 172.21.15.153:58570) 2024-03-11T23:00:29.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.725+0000 7f9287fff640 1 -- 172.21.15.153:0/3705495728 learned_addr learned my addr 172.21.15.153:0/3705495728 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T23:00:29.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.725+0000 7f928cab9640 1 --2- 172.21.15.153:0/3705495728 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92880fb180 0x7f928810cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:29.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.725+0000 7f9287fff640 1 -- 172.21.15.153:0/3705495728 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92880fb180 msgr2=0x7f928810cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:29.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.725+0000 7f9287fff640 1 --2- 172.21.15.153:0/3705495728 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92880fb180 0x7f928810cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:29.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.725+0000 7f9287fff640 1 -- 172.21.15.153:0/3705495728 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f927c005040 con 0x7f92880fc450 2024-03-11T23:00:29.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.725+0000 7f9287fff640 1 --2- 172.21.15.153:0/3705495728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92880fc450 0x7f928810d0a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f927400eb70 tx=0x7f927400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:00:29.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.725+0000 7f92857fa640 1 -- 172.21.15.153:0/3705495728 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9274007c30 con 0x7f92880fc450 2024-03-11T23:00:29.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.725+0000 7f92857fa640 1 -- 172.21.15.153:0/3705495728 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f927400f040 con 0x7f92880fc450 2024-03-11T23:00:29.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.725+0000 7f928ed1d640 1 -- 172.21.15.153:0/3705495728 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9288112280 con 0x7f92880fc450 2024-03-11T23:00:29.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.725+0000 7f92857fa640 1 -- 172.21.15.153:0/3705495728 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f92740095b0 con 0x7f92880fc450 2024-03-11T23:00:29.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.725+0000 7f928ed1d640 1 -- 172.21.15.153:0/3705495728 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9288112770 con 0x7f92880fc450 2024-03-11T23:00:29.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.726+0000 7f928ed1d640 1 -- 172.21.15.153:0/3705495728 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9288103cf0 con 0x7f92880fc450 2024-03-11T23:00:29.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.727+0000 7f92857fa640 1 -- 172.21.15.153:0/3705495728 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f9274009730 con 0x7f92880fc450 2024-03-11T23:00:29.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.727+0000 7f92857fa640 1 --2- 172.21.15.153:0/3705495728 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f9278069210 0x7f927806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:29.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.727+0000 7f92857fa640 1 -- 172.21.15.153:0/3705495728 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f9274086eb0 con 0x7f92880fc450 2024-03-11T23:00:29.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.727+0000 7f928cab9640 1 --2- 172.21.15.153:0/3705495728 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f9278069210 0x7f927806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:29.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.728+0000 7f928cab9640 1 --2- 172.21.15.153:0/3705495728 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f9278069210 0x7f927806b6c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f927c000c00 tx=0x7f927c009da0 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:00:29.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.732+0000 7f92857fa640 1 -- 172.21.15.153:0/3705495728 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f92740531c0 con 0x7f92880fc450 2024-03-11T23:00:29.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:29 smithi163 ceph-mon[35584]: pgmap v190: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-11T23:00:29.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:29 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:29.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:29 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:29.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:29 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:00:29.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:29 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:00:29.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:29 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:29.739 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:29 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/4266776863' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-03-11T23:00:29.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.899+0000 7f928ed1d640 1 -- 172.21.15.153:0/3705495728 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9288112ad0 con 0x7f92880fc450 2024-03-11T23:00:29.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.900+0000 7f92857fa640 1 -- 172.21.15.153:0/3705495728 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+14462 (secure 0 0 0) 0x7f92740567e0 con 0x7f92880fc450 2024-03-11T23:00:29.904 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T23:00:29.904 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":47,"fsid":"fa913b96-dff9-11ee-95c8-87774f69a715","created":"2024-03-11T22:53:14.269171+0000","modified":"2024-03-11T23:00:14.140527+0000","last_up_change":"2024-03-11T22:58:40.147211+0000","last_in_change":"2024-03-11T22:58:21.268596+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T22:57:26.368808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-03-11T22:59:58.162442+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-03-11T22:59:58.690349+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"542731eb-3fb3-450b-93d6-c6a8a9df1659","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1406194697},{"type":"v1","addr":"172.21.15.153:6803","nonce":1406194697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1406194697},{"type":"v1","addr":"172.21.15.153:6805","nonce":1406194697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1406194697},{"type":"v1","addr":"172.21.15.153:6809","nonce":1406194697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1406194697},{"type":"v1","addr":"172.21.15.153:6807","nonce":1406194697}]},"public_addr":"172.21.15.153:6803/1406194697","cluster_addr":"172.21.15.153:6805/1406194697","heartbeat_back_addr":"172.21.15.153:6809/1406194697","heartbeat_front_addr":"172.21.15.153:6807/1406194697","state":["exists","up"]},{"osd":1,"uuid":"7be06d22-d83a-4b39-ace8-3d800bbd1aee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":2718652372},{"type":"v1","addr":"172.21.15.153:6811","nonce":2718652372}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":2718652372},{"type":"v1","addr":"172.21.15.153:6813","nonce":2718652372}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":2718652372},{"type":"v1","addr":"172.21.15.153:6817","nonce":2718652372}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":2718652372},{"type":"v1","addr":"172.21.15.153:6815","nonce":2718652372}]},"public_addr":"172.21.15.153:6811/2718652372","cluster_addr":"172.21.15.153:6813/2718652372","heartbeat_back_addr":"172.21.15.153:6817/2718652372","heartbeat_front_addr":"172.21.15.153:6815/2718652372","state":["exists","up"]},{"osd":2,"uuid":"b4863162-3a3d-45ae-9bc0-c8b069ff063a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":4016153774},{"type":"v1","addr":"172.21.15.153:6819","nonce":4016153774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":4016153774},{"type":"v1","addr":"172.21.15.153:6821","nonce":4016153774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":4016153774},{"type":"v1","addr":"172.21.15.153:6825","nonce":4016153774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":4016153774},{"type":"v1","addr":"172.21.15.153:6823","nonce":4016153774}]},"public_addr":"172.21.15.153:6819/4016153774","cluster_addr":"172.21.15.153:6821/4016153774","heartbeat_back_addr":"172.21.15.153:6825/4016153774","heartbeat_front_addr":"172.21.15.153:6823/4016153774","state":["exists","up"]},{"osd":3,"uuid":"92a62b43-e245-4d71-b4ca-89a7d518af50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":3768791983},{"type":"v1","addr":"172.21.15.163:6801","nonce":3768791983}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":3768791983},{"type":"v1","addr":"172.21.15.163:6803","nonce":3768791983}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":3768791983},{"type":"v1","addr":"172.21.15.163:6807","nonce":3768791983}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":3768791983},{"type":"v1","addr":"172.21.15.163:6805","nonce":3768791983}]},"public_addr":"172.21.15.163:6801/3768791983","cluster_addr":"172.21.15.163:6803/3768791983","heartbeat_back_addr":"172.21.15.163:6807/3768791983","heartbeat_front_addr":"172.21.15.163:6805/3768791983","state":["exists","up"]},{"osd":4,"uuid":"b9caf0a9-46bf-44bc-a2d0-b5f0e1b3eadf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":4243752277},{"type":"v1","addr":"172.21.15.163:6809","nonce":4243752277}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":4243752277},{"type":"v1","addr":"172.21.15.163:6811","nonce":4243752277}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":4243752277},{"type":"v1","addr":"172.21.15.163:6815","nonce":4243752277}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":4243752277},{"type":"v1","addr":"172.21.15.163:6813","nonce":4243752277}]},"public_addr":"172.21.15.163:6809/4243752277","cluster_addr":"172.21.15.163:6811/4243752277","heartbeat_back_addr":"172.21.15.163:6815/4243752277","heartbeat_front_addr":"172.21.15.163:6813/4243752277","state":["exists","up"]},{"osd":5,"uuid":"b4e2dc8c-98c7-4be5-a0a2-917ec54e5910","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2486862776},{"type":"v1","addr":"172.21.15.163:6817","nonce":2486862776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2486862776},{"type":"v1","addr":"172.21.15.163:6819","nonce":2486862776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2486862776},{"type":"v1","addr":"172.21.15.163:6823","nonce":2486862776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2486862776},{"type":"v1","addr":"172.21.15.163:6821","nonce":2486862776}]},"public_addr":"172.21.15.163:6817/2486862776","cluster_addr":"172.21.15.163:6819/2486862776","heartbeat_back_addr":"172.21.15.163:6823/2486862776","heartbeat_front_addr":"172.21.15.163:6821/2486862776","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:56:27.613880+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:56:55.525968+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:57:22.785587+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:57:47.270057+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:58:12.791831+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:58:38.648113+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.163:6824/3959509309":"2024-03-12T23:00:14.140490+0000","172.21.15.153:0/3612064784":"2024-03-12T22:53:46.835889+0000","172.21.15.153:0/3414319920":"2024-03-12T22:54:17.065945+0000","172.21.15.153:0/3587313628":"2024-03-12T22:53:46.835889+0000","172.21.15.153:6801/2578439373":"2024-03-12T22:53:46.835889+0000","172.21.15.153:0/2653190270":"2024-03-12T22:54:17.065945+0000","172.21.15.153:6800/2578439373":"2024-03-12T22:53:46.835889+0000","172.21.15.153:0/1974568910":"2024-03-12T22:54:17.065945+0000","172.21.15.153:6800/1518383615":"2024-03-12T22:54:17.065945+0000","172.21.15.153:6801/1518383615":"2024-03-12T22:54:17.065945+0000","172.21.15.153:6800/4212724556":"2024-03-12T22:55:18.043211+0000","172.21.15.153:0/369153497":"2024-03-12T22:55:18.043211+0000","172.21.15.153:6801/4212724556":"2024-03-12T22:55:18.043211+0000","172.21.15.163:6825/3959509309":"2024-03-12T23:00:14.140490+0000","172.21.15.153:0/41949006":"2024-03-12T22:55:18.043211+0000","172.21.15.153:0/3032487131":"2024-03-12T22:53:46.835889+0000","172.21.15.153:0/1864158862":"2024-03-12T22:55:18.043211+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T23:00:29.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.902+0000 7f928ed1d640 1 -- 172.21.15.153:0/3705495728 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f9278069210 msgr2=0x7f927806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:29.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.902+0000 7f928ed1d640 1 --2- 172.21.15.153:0/3705495728 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f9278069210 0x7f927806b6c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f927c000c00 tx=0x7f927c009da0 comp rx=0 tx=0).stop 2024-03-11T23:00:29.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.902+0000 7f928ed1d640 1 -- 172.21.15.153:0/3705495728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92880fc450 msgr2=0x7f928810d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:29.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.902+0000 7f928ed1d640 1 --2- 172.21.15.153:0/3705495728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92880fc450 0x7f928810d0a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f927400eb70 tx=0x7f927400b040 comp rx=0 tx=0).stop 2024-03-11T23:00:29.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.902+0000 7f928ed1d640 1 -- 172.21.15.153:0/3705495728 shutdown_connections 2024-03-11T23:00:29.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.902+0000 7f928ed1d640 1 --2- 172.21.15.153:0/3705495728 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f9278069210 0x7f927806b6c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:29.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.902+0000 7f928ed1d640 1 --2- 172.21.15.153:0/3705495728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92880fc450 0x7f928810d0a0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:29.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.902+0000 7f928ed1d640 1 --2- 172.21.15.153:0/3705495728 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f92880fb180 0x7f928810cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:29.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.902+0000 7f928ed1d640 1 -- 172.21.15.153:0/3705495728 >> 172.21.15.153:0/3705495728 conn(0x7f92880f6940 msgr2=0x7f92880656f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:00:29.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.902+0000 7f928ed1d640 1 -- 172.21.15.153:0/3705495728 shutdown_connections 2024-03-11T23:00:29.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:29.902+0000 7f928ed1d640 1 -- 172.21.15.153:0/3705495728 wait complete. 2024-03-11T23:00:29.917 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T22:57:26.368808+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-03-11T22:59:58.162442+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-03-11T22:59:58.690349+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-11T23:00:29.918 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-11T23:00:30.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.014+0000 7feeced9d640 1 -- 172.21.15.153:0/2358089358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feec80fc2c0 msgr2=0x7feec80fc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:30.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.014+0000 7feeced9d640 1 --2- 172.21.15.153:0/2358089358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feec80fc2c0 0x7feec80fc6d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7feeb80048d0 tx=0x7feeb8009da0 comp rx=0 tx=0).stop 2024-03-11T23:00:30.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.014+0000 7feeced9d640 1 -- 172.21.15.153:0/2358089358 shutdown_connections 2024-03-11T23:00:30.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.014+0000 7feeced9d640 1 --2- 172.21.15.153:0/2358089358 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feec80fd590 0x7feec80fda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:30.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.014+0000 7feeced9d640 1 --2- 172.21.15.153:0/2358089358 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feec80fc2c0 0x7feec80fc6d0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:30.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.014+0000 7feeced9d640 1 -- 172.21.15.153:0/2358089358 >> 172.21.15.153:0/2358089358 conn(0x7feec80f7a60 msgr2=0x7feec80f9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:00:30.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.014+0000 7feeced9d640 1 -- 172.21.15.153:0/2358089358 shutdown_connections 2024-03-11T23:00:30.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.014+0000 7feeced9d640 1 -- 172.21.15.153:0/2358089358 wait complete. 2024-03-11T23:00:30.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.015+0000 7feeced9d640 1 Processor -- start 2024-03-11T23:00:30.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.015+0000 7feeced9d640 1 -- start start 2024-03-11T23:00:30.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.015+0000 7feeced9d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feec80fc2c0 0x7feec810cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:30.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.015+0000 7feeced9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feec80fd590 0x7feec810d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:30.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.015+0000 7feeced9d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec810d6e0 con 0x7feec80fd590 2024-03-11T23:00:30.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.015+0000 7feeced9d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec8112130 con 0x7feec80fc2c0 2024-03-11T23:00:30.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.015+0000 7feec7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feec80fd590 0x7feec810d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:30.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.015+0000 7feec7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feec80fd590 0x7feec810d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:58588/0 (socket says 172.21.15.153:58588) 2024-03-11T23:00:30.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.015+0000 7feec7fff640 1 -- 172.21.15.153:0/3350537991 learned_addr learned my addr 172.21.15.153:0/3350537991 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T23:00:30.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.015+0000 7feeccb39640 1 --2- 172.21.15.153:0/3350537991 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feec80fc2c0 0x7feec810cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:30.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.016+0000 7feec7fff640 1 -- 172.21.15.153:0/3350537991 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feec80fc2c0 msgr2=0x7feec810cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:30.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.016+0000 7feec7fff640 1 --2- 172.21.15.153:0/3350537991 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feec80fc2c0 0x7feec810cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:30.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.016+0000 7feec7fff640 1 -- 172.21.15.153:0/3350537991 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeb8005040 con 0x7feec80fd590 2024-03-11T23:00:30.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.016+0000 7feec7fff640 1 --2- 172.21.15.153:0/3350537991 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feec80fd590 0x7feec810d090 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7feebc002a50 tx=0x7feebc004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:00:30.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.016+0000 7feec57fa640 1 -- 172.21.15.153:0/3350537991 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feebc007c30 con 0x7feec80fd590 2024-03-11T23:00:30.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.016+0000 7feec57fa640 1 -- 172.21.15.153:0/3350537991 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feebc00f040 con 0x7feec80fd590 2024-03-11T23:00:30.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.016+0000 7feeced9d640 1 -- 172.21.15.153:0/3350537991 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feec81122d0 con 0x7feec80fd590 2024-03-11T23:00:30.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.016+0000 7feec57fa640 1 -- 172.21.15.153:0/3350537991 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feebc00db50 con 0x7feec80fd590 2024-03-11T23:00:30.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.016+0000 7feeced9d640 1 -- 172.21.15.153:0/3350537991 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feec81127c0 con 0x7feec80fd590 2024-03-11T23:00:30.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.017+0000 7feeced9d640 1 -- 172.21.15.153:0/3350537991 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feec8104e30 con 0x7feec80fd590 2024-03-11T23:00:30.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.017+0000 7feec57fa640 1 -- 172.21.15.153:0/3350537991 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7feebc01c400 con 0x7feec80fd590 2024-03-11T23:00:30.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.018+0000 7feec57fa640 1 --2- 172.21.15.153:0/3350537991 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7feeb0069210 0x7feeb006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:30.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.018+0000 7feeccb39640 1 --2- 172.21.15.153:0/3350537991 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7feeb0069210 0x7feeb006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:30.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.018+0000 7feec57fa640 1 -- 172.21.15.153:0/3350537991 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7feebc086ad0 con 0x7feec80fd590 2024-03-11T23:00:30.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.018+0000 7feeccb39640 1 --2- 172.21.15.153:0/3350537991 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7feeb0069210 0x7feeb006b6c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7feeb8000c00 tx=0x7feeb8009da0 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:00:30.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.022+0000 7feec57fa640 1 -- 172.21.15.153:0/3350537991 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7feebc04cd20 con 0x7feec80fd590 2024-03-11T23:00:30.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.186+0000 7feeced9d640 1 -- 172.21.15.153:0/3350537991 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7feec81129e0 con 0x7feec80fd590 2024-03-11T23:00:30.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.187+0000 7feec57fa640 1 -- 172.21.15.153:0/3350537991 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v47) v1 ==== 93+0+10 (secure 0 0 0) 0x7feebc052d30 con 0x7feec80fd590 2024-03-11T23:00:30.191 INFO:teuthology.orchestra.run.smithi153.stdout:pg_num: 1 2024-03-11T23:00:30.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.189+0000 7feeced9d640 1 -- 172.21.15.153:0/3350537991 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7feeb0069210 msgr2=0x7feeb006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:30.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.189+0000 7feeced9d640 1 --2- 172.21.15.153:0/3350537991 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7feeb0069210 0x7feeb006b6c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7feeb8000c00 tx=0x7feeb8009da0 comp rx=0 tx=0).stop 2024-03-11T23:00:30.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.189+0000 7feeced9d640 1 -- 172.21.15.153:0/3350537991 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feec80fd590 msgr2=0x7feec810d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:30.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.189+0000 7feeced9d640 1 --2- 172.21.15.153:0/3350537991 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feec80fd590 0x7feec810d090 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7feebc002a50 tx=0x7feebc004600 comp rx=0 tx=0).stop 2024-03-11T23:00:30.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.189+0000 7feeced9d640 1 -- 172.21.15.153:0/3350537991 shutdown_connections 2024-03-11T23:00:30.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.189+0000 7feeced9d640 1 --2- 172.21.15.153:0/3350537991 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7feeb0069210 0x7feeb006b6c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:30.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.189+0000 7feeced9d640 1 --2- 172.21.15.153:0/3350537991 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feec80fd590 0x7feec810d090 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:30.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.189+0000 7feeced9d640 1 --2- 172.21.15.153:0/3350537991 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7feec80fc2c0 0x7feec810cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:30.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.189+0000 7feeced9d640 1 -- 172.21.15.153:0/3350537991 >> 172.21.15.153:0/3350537991 conn(0x7feec80f7a60 msgr2=0x7feec81007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:00:30.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.189+0000 7feeced9d640 1 -- 172.21.15.153:0/3350537991 shutdown_connections 2024-03-11T23:00:30.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.189+0000 7feeced9d640 1 -- 172.21.15.153:0/3350537991 wait complete. 2024-03-11T23:00:30.205 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-03-11T23:00:30.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.301+0000 7fc46286e640 1 -- 172.21.15.153:0/421679349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc45c0fb180 msgr2=0x7fc45c0fb590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:30.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.301+0000 7fc46286e640 1 --2- 172.21.15.153:0/421679349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc45c0fb180 0x7fc45c0fb590 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc44c0048d0 tx=0x7fc44c009da0 comp rx=0 tx=0).stop 2024-03-11T23:00:30.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.301+0000 7fc46286e640 1 -- 172.21.15.153:0/421679349 shutdown_connections 2024-03-11T23:00:30.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.301+0000 7fc46286e640 1 --2- 172.21.15.153:0/421679349 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc45c0fc450 0x7fc45c0fc8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:30.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.301+0000 7fc46286e640 1 --2- 172.21.15.153:0/421679349 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc45c0fb180 0x7fc45c0fb590 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:30.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.301+0000 7fc46286e640 1 -- 172.21.15.153:0/421679349 >> 172.21.15.153:0/421679349 conn(0x7fc45c0f6960 msgr2=0x7fc45c0f8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:00:30.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.301+0000 7fc46286e640 1 -- 172.21.15.153:0/421679349 shutdown_connections 2024-03-11T23:00:30.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.301+0000 7fc46286e640 1 -- 172.21.15.153:0/421679349 wait complete. 2024-03-11T23:00:30.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.302+0000 7fc46286e640 1 Processor -- start 2024-03-11T23:00:30.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.302+0000 7fc46286e640 1 -- start start 2024-03-11T23:00:30.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.302+0000 7fc46286e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc45c0fb180 0x7fc45c066140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:30.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.302+0000 7fc46286e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc45c0fc450 0x7fc45c111930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:30.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.302+0000 7fc46286e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c111e70 con 0x7fc45c0fc450 2024-03-11T23:00:30.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.302+0000 7fc46286e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c111fb0 con 0x7fc45c0fb180 2024-03-11T23:00:30.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.302+0000 7fc45b7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc45c0fc450 0x7fc45c111930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:30.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.302+0000 7fc45b7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc45c0fc450 0x7fc45c111930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:58618/0 (socket says 172.21.15.153:58618) 2024-03-11T23:00:30.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.302+0000 7fc45b7fe640 1 -- 172.21.15.153:0/3930979411 learned_addr learned my addr 172.21.15.153:0/3930979411 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T23:00:30.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.302+0000 7fc45bfff640 1 --2- 172.21.15.153:0/3930979411 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc45c0fb180 0x7fc45c066140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:30.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.302+0000 7fc45b7fe640 1 -- 172.21.15.153:0/3930979411 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc45c0fb180 msgr2=0x7fc45c066140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:30.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.302+0000 7fc45b7fe640 1 --2- 172.21.15.153:0/3930979411 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc45c0fb180 0x7fc45c066140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:30.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.302+0000 7fc45b7fe640 1 -- 172.21.15.153:0/3930979411 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc44c005040 con 0x7fc45c0fc450 2024-03-11T23:00:30.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.303+0000 7fc45b7fe640 1 --2- 172.21.15.153:0/3930979411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc45c0fc450 0x7fc45c111930 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc45000eb70 tx=0x7fc45000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:00:30.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.303+0000 7fc458ff9640 1 -- 172.21.15.153:0/3930979411 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc450007c30 con 0x7fc45c0fc450 2024-03-11T23:00:30.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.303+0000 7fc46286e640 1 -- 172.21.15.153:0/3930979411 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc45c1121e0 con 0x7fc45c0fc450 2024-03-11T23:00:30.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.303+0000 7fc458ff9640 1 -- 172.21.15.153:0/3930979411 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc45000f040 con 0x7fc45c0fc450 2024-03-11T23:00:30.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.303+0000 7fc46286e640 1 -- 172.21.15.153:0/3930979411 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc45c112700 con 0x7fc45c0fc450 2024-03-11T23:00:30.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.303+0000 7fc458ff9640 1 -- 172.21.15.153:0/3930979411 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc4500095f0 con 0x7fc45c0fc450 2024-03-11T23:00:30.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.304+0000 7fc46286e640 1 -- 172.21.15.153:0/3930979411 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc45c103cd0 con 0x7fc45c0fc450 2024-03-11T23:00:30.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.304+0000 7fc458ff9640 1 -- 172.21.15.153:0/3930979411 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fc45000ccc0 con 0x7fc45c0fc450 2024-03-11T23:00:30.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.305+0000 7fc458ff9640 1 --2- 172.21.15.153:0/3930979411 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fc444069220 0x7fc44406b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:30.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.305+0000 7fc458ff9640 1 -- 172.21.15.153:0/3930979411 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fc450086e40 con 0x7fc45c0fc450 2024-03-11T23:00:30.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.305+0000 7fc45bfff640 1 --2- 172.21.15.153:0/3930979411 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fc444069220 0x7fc44406b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:30.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.305+0000 7fc45bfff640 1 --2- 172.21.15.153:0/3930979411 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fc444069220 0x7fc44406b6d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc44c034040 tx=0x7fc44c009da0 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:00:30.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.309+0000 7fc458ff9640 1 -- 172.21.15.153:0/3930979411 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7fc45004dd20 con 0x7fc45c0fc450 2024-03-11T23:00:30.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.471+0000 7fc46286e640 1 -- 172.21.15.153:0/3930979411 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7fc45c112a30 con 0x7fc45c0fc450 2024-03-11T23:00:30.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.472+0000 7fc458ff9640 1 -- 172.21.15.153:0/3930979411 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v47) v1 ==== 107+0+11 (secure 0 0 0) 0x7fc450053150 con 0x7fc45c0fc450 2024-03-11T23:00:30.476 INFO:teuthology.orchestra.run.smithi153.stdout:pg_num: 32 2024-03-11T23:00:30.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.474+0000 7fc46286e640 1 -- 172.21.15.153:0/3930979411 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fc444069220 msgr2=0x7fc44406b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:30.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.474+0000 7fc46286e640 1 --2- 172.21.15.153:0/3930979411 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fc444069220 0x7fc44406b6d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc44c034040 tx=0x7fc44c009da0 comp rx=0 tx=0).stop 2024-03-11T23:00:30.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.474+0000 7fc46286e640 1 -- 172.21.15.153:0/3930979411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc45c0fc450 msgr2=0x7fc45c111930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:30.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.474+0000 7fc46286e640 1 --2- 172.21.15.153:0/3930979411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc45c0fc450 0x7fc45c111930 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc45000eb70 tx=0x7fc45000b040 comp rx=0 tx=0).stop 2024-03-11T23:00:30.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.474+0000 7fc46286e640 1 -- 172.21.15.153:0/3930979411 shutdown_connections 2024-03-11T23:00:30.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.474+0000 7fc46286e640 1 --2- 172.21.15.153:0/3930979411 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fc444069220 0x7fc44406b6d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:30.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.474+0000 7fc46286e640 1 --2- 172.21.15.153:0/3930979411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc45c0fc450 0x7fc45c111930 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:30.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.474+0000 7fc46286e640 1 --2- 172.21.15.153:0/3930979411 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc45c0fb180 0x7fc45c066140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:30.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.474+0000 7fc46286e640 1 -- 172.21.15.153:0/3930979411 >> 172.21.15.153:0/3930979411 conn(0x7fc45c0f6960 msgr2=0x7fc45c0ff680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:00:30.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.474+0000 7fc46286e640 1 -- 172.21.15.153:0/3930979411 shutdown_connections 2024-03-11T23:00:30.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.474+0000 7fc46286e640 1 -- 172.21.15.153:0/3930979411 wait complete. 2024-03-11T23:00:30.488 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:30 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/3705495728' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T23:00:30.488 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:30 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/3350537991' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T23:00:30.489 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-03-11T23:00:30.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.585+0000 7f2b9b66e640 1 -- 172.21.15.153:0/4260219176 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b940fd5e0 msgr2=0x7f2b940fda50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:30.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.585+0000 7f2b9b66e640 1 --2- 172.21.15.153:0/4260219176 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b940fd5e0 0x7f2b940fda50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2b840048d0 tx=0x7f2b84009da0 comp rx=0 tx=0).stop 2024-03-11T23:00:30.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.585+0000 7f2b9b66e640 1 -- 172.21.15.153:0/4260219176 shutdown_connections 2024-03-11T23:00:30.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.585+0000 7f2b9b66e640 1 --2- 172.21.15.153:0/4260219176 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b940fd5e0 0x7f2b940fda50 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:30.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.585+0000 7f2b9b66e640 1 --2- 172.21.15.153:0/4260219176 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b940fa5d0 0x7f2b940fa9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:30.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.585+0000 7f2b9b66e640 1 -- 172.21.15.153:0/4260219176 >> 172.21.15.153:0/4260219176 conn(0x7f2b940f5ff0 msgr2=0x7f2b940f8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:00:30.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.586+0000 7f2b9b66e640 1 -- 172.21.15.153:0/4260219176 shutdown_connections 2024-03-11T23:00:30.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.586+0000 7f2b9b66e640 1 -- 172.21.15.153:0/4260219176 wait complete. 2024-03-11T23:00:30.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.586+0000 7f2b9b66e640 1 Processor -- start 2024-03-11T23:00:30.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.586+0000 7f2b9b66e640 1 -- start start 2024-03-11T23:00:30.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.586+0000 7f2b9b66e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b940fa5d0 0x7f2b9410cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:30.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.586+0000 7f2b9b66e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b940fd5e0 0x7f2b9410d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:30.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.586+0000 7f2b9b66e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9410d720 con 0x7f2b940fd5e0 2024-03-11T23:00:30.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.586+0000 7f2b9b66e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b94112170 con 0x7f2b940fa5d0 2024-03-11T23:00:30.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.586+0000 7f2b98c09640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b940fd5e0 0x7f2b9410d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:30.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.586+0000 7f2b98c09640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b940fd5e0 0x7f2b9410d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:58630/0 (socket says 172.21.15.153:58630) 2024-03-11T23:00:30.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.586+0000 7f2b98c09640 1 -- 172.21.15.153:0/172910397 learned_addr learned my addr 172.21.15.153:0/172910397 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T23:00:30.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.586+0000 7f2b9940a640 1 --2- 172.21.15.153:0/172910397 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b940fa5d0 0x7f2b9410cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:30.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.586+0000 7f2b98c09640 1 -- 172.21.15.153:0/172910397 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b940fa5d0 msgr2=0x7f2b9410cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:30.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.587+0000 7f2b98c09640 1 --2- 172.21.15.153:0/172910397 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b940fa5d0 0x7f2b9410cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:30.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.587+0000 7f2b98c09640 1 -- 172.21.15.153:0/172910397 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b84005040 con 0x7f2b940fd5e0 2024-03-11T23:00:30.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.587+0000 7f2b98c09640 1 --2- 172.21.15.153:0/172910397 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b940fd5e0 0x7f2b9410d0d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2b84034040 tx=0x7f2b84009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:00:30.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.587+0000 7f2b89ffb640 1 -- 172.21.15.153:0/172910397 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b84009590 con 0x7f2b940fd5e0 2024-03-11T23:00:30.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.587+0000 7f2b9b66e640 1 -- 172.21.15.153:0/172910397 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b941122b0 con 0x7f2b940fd5e0 2024-03-11T23:00:30.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.587+0000 7f2b89ffb640 1 -- 172.21.15.153:0/172910397 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2b840096f0 con 0x7f2b940fd5e0 2024-03-11T23:00:30.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.587+0000 7f2b89ffb640 1 -- 172.21.15.153:0/172910397 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2b8403b6b0 con 0x7f2b940fd5e0 2024-03-11T23:00:30.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.587+0000 7f2b9b66e640 1 -- 172.21.15.153:0/172910397 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b941127a0 con 0x7f2b940fd5e0 2024-03-11T23:00:30.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.588+0000 7f2b9b66e640 1 -- 172.21.15.153:0/172910397 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b94104e50 con 0x7f2b940fd5e0 2024-03-11T23:00:30.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.588+0000 7f2b89ffb640 1 -- 172.21.15.153:0/172910397 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f2b8402f070 con 0x7f2b940fd5e0 2024-03-11T23:00:30.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.589+0000 7f2b89ffb640 1 --2- 172.21.15.153:0/172910397 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f2b80069210 0x7f2b8006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:30.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.589+0000 7f2b9940a640 1 --2- 172.21.15.153:0/172910397 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f2b80069210 0x7f2b8006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:30.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.589+0000 7f2b89ffb640 1 -- 172.21.15.153:0/172910397 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f2b840a98e0 con 0x7f2b940fd5e0 2024-03-11T23:00:30.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.589+0000 7f2b9940a640 1 --2- 172.21.15.153:0/172910397 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f2b80069210 0x7f2b8006b6c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2b94065410 tx=0x7f2b90009040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:00:30.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.593+0000 7f2b89ffb640 1 -- 172.21.15.153:0/172910397 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f2b84075b40 con 0x7f2b940fd5e0 2024-03-11T23:00:30.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:30 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/3705495728' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T23:00:30.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:30 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/3350537991' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T23:00:30.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.756+0000 7f2b9b66e640 1 -- 172.21.15.153:0/172910397 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f2b94112a80 con 0x7f2b940fd5e0 2024-03-11T23:00:30.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.756+0000 7f2b89ffb640 1 -- 172.21.15.153:0/172910397 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v47) v1 ==== 107+0+11 (secure 0 0 0) 0x7f2b84079160 con 0x7f2b940fd5e0 2024-03-11T23:00:30.760 INFO:teuthology.orchestra.run.smithi153.stdout:pg_num: 32 2024-03-11T23:00:30.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.758+0000 7f2b9b66e640 1 -- 172.21.15.153:0/172910397 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f2b80069210 msgr2=0x7f2b8006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:30.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.758+0000 7f2b9b66e640 1 --2- 172.21.15.153:0/172910397 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f2b80069210 0x7f2b8006b6c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2b94065410 tx=0x7f2b90009040 comp rx=0 tx=0).stop 2024-03-11T23:00:30.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.758+0000 7f2b9b66e640 1 -- 172.21.15.153:0/172910397 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b940fd5e0 msgr2=0x7f2b9410d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:30.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.758+0000 7f2b9b66e640 1 --2- 172.21.15.153:0/172910397 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b940fd5e0 0x7f2b9410d0d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2b84034040 tx=0x7f2b84009da0 comp rx=0 tx=0).stop 2024-03-11T23:00:30.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.758+0000 7f2b9b66e640 1 -- 172.21.15.153:0/172910397 shutdown_connections 2024-03-11T23:00:30.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.758+0000 7f2b9b66e640 1 --2- 172.21.15.153:0/172910397 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f2b80069210 0x7f2b8006b6c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:30.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.758+0000 7f2b9b66e640 1 --2- 172.21.15.153:0/172910397 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2b940fd5e0 0x7f2b9410d0d0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:30.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.758+0000 7f2b9b66e640 1 --2- 172.21.15.153:0/172910397 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2b940fa5d0 0x7f2b9410cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:30.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.758+0000 7f2b9b66e640 1 -- 172.21.15.153:0/172910397 >> 172.21.15.153:0/172910397 conn(0x7f2b940f5ff0 msgr2=0x7f2b94100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:00:30.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.758+0000 7f2b9b66e640 1 -- 172.21.15.153:0/172910397 shutdown_connections 2024-03-11T23:00:30.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.758+0000 7f2b9b66e640 1 -- 172.21.15.153:0/172910397 wait complete. 2024-03-11T23:00:30.775 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-11T23:00:30.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.868+0000 7f52a2c6e640 1 -- 172.21.15.153:0/3605316864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f529c0fd4d0 msgr2=0x7f529c0fd940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:30.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.868+0000 7f52a2c6e640 1 --2- 172.21.15.153:0/3605316864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f529c0fd4d0 0x7f529c0fd940 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f52900048d0 tx=0x7f5290009da0 comp rx=0 tx=0).stop 2024-03-11T23:00:30.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.868+0000 7f52a2c6e640 1 -- 172.21.15.153:0/3605316864 shutdown_connections 2024-03-11T23:00:30.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.868+0000 7f52a2c6e640 1 --2- 172.21.15.153:0/3605316864 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f529c0fd4d0 0x7f529c0fd940 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:30.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.868+0000 7f52a2c6e640 1 --2- 172.21.15.153:0/3605316864 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f529c0fc2a0 0x7f529c0fc6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:30.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.868+0000 7f52a2c6e640 1 -- 172.21.15.153:0/3605316864 >> 172.21.15.153:0/3605316864 conn(0x7f529c0f7980 msgr2=0x7f529c0f9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:00:30.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.868+0000 7f52a2c6e640 1 -- 172.21.15.153:0/3605316864 shutdown_connections 2024-03-11T23:00:30.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.868+0000 7f52a2c6e640 1 -- 172.21.15.153:0/3605316864 wait complete. 2024-03-11T23:00:30.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.869+0000 7f52a2c6e640 1 Processor -- start 2024-03-11T23:00:30.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.869+0000 7f52a2c6e640 1 -- start start 2024-03-11T23:00:30.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.869+0000 7f52a2c6e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f529c0fc2a0 0x7f529c10cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:30.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.869+0000 7f52a2c6e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f529c0fd4d0 0x7f529c10d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:30.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.869+0000 7f52a2c6e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f529c10d6c0 con 0x7f529c0fc2a0 2024-03-11T23:00:30.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.869+0000 7f52a2c6e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f529c112110 con 0x7f529c0fd4d0 2024-03-11T23:00:30.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.869+0000 7f52a0a0a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f529c0fc2a0 0x7f529c10cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:30.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.869+0000 7f52a0a0a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f529c0fc2a0 0x7f529c10cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:58638/0 (socket says 172.21.15.153:58638) 2024-03-11T23:00:30.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.869+0000 7f52a0a0a640 1 -- 172.21.15.153:0/2944395128 learned_addr learned my addr 172.21.15.153:0/2944395128 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T23:00:30.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.869+0000 7f529bfff640 1 --2- 172.21.15.153:0/2944395128 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f529c0fd4d0 0x7f529c10d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:30.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.869+0000 7f52a0a0a640 1 -- 172.21.15.153:0/2944395128 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f529c0fd4d0 msgr2=0x7f529c10d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:30.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.869+0000 7f52a0a0a640 1 --2- 172.21.15.153:0/2944395128 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f529c0fd4d0 0x7f529c10d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:30.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.869+0000 7f52a0a0a640 1 -- 172.21.15.153:0/2944395128 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5290005040 con 0x7f529c0fc2a0 2024-03-11T23:00:30.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.870+0000 7f52a0a0a640 1 --2- 172.21.15.153:0/2944395128 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f529c0fc2a0 0x7f529c10cb30 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f528c002840 tx=0x7f528c004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:00:30.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.870+0000 7f52997fa640 1 -- 172.21.15.153:0/2944395128 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f528c007c30 con 0x7f529c0fc2a0 2024-03-11T23:00:30.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.870+0000 7f52a2c6e640 1 -- 172.21.15.153:0/2944395128 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f529c1122b0 con 0x7f529c0fc2a0 2024-03-11T23:00:30.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.870+0000 7f52997fa640 1 -- 172.21.15.153:0/2944395128 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f528c00f040 con 0x7f529c0fc2a0 2024-03-11T23:00:30.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.870+0000 7f52997fa640 1 -- 172.21.15.153:0/2944395128 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f528c009570 con 0x7f529c0fc2a0 2024-03-11T23:00:30.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.870+0000 7f52a2c6e640 1 -- 172.21.15.153:0/2944395128 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f529c1127a0 con 0x7f529c0fc2a0 2024-03-11T23:00:30.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.871+0000 7f52a2c6e640 1 -- 172.21.15.153:0/2944395128 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f529c104e10 con 0x7f529c0fc2a0 2024-03-11T23:00:30.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.871+0000 7f52997fa640 1 -- 172.21.15.153:0/2944395128 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f528c00dc80 con 0x7f529c0fc2a0 2024-03-11T23:00:30.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.872+0000 7f52997fa640 1 --2- 172.21.15.153:0/2944395128 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f5284069210 0x7f528406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:30.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.872+0000 7f52997fa640 1 -- 172.21.15.153:0/2944395128 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f528c0871a0 con 0x7f529c0fc2a0 2024-03-11T23:00:30.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.872+0000 7f529bfff640 1 --2- 172.21.15.153:0/2944395128 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f5284069210 0x7f528406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:30.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.872+0000 7f529bfff640 1 --2- 172.21.15.153:0/2944395128 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f5284069210 0x7f528406b6c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f529c10e080 tx=0x7f5290009da0 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:00:30.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:30.876+0000 7f52997fa640 1 -- 172.21.15.153:0/2944395128 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f528c053400 con 0x7f529c0fc2a0 2024-03-11T23:00:31.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:31.077+0000 7f52a2c6e640 1 -- 172.21.15.153:0/2944395128 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f529c112c40 con 0x7f529c0fc2a0 2024-03-11T23:00:31.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:31.077+0000 7f52997fa640 1 -- 172.21.15.153:0/2944395128 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+4725 (secure 0 0 0) 0x7f528c056a20 con 0x7f529c0fc2a0 2024-03-11T23:00:31.082 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-11T23:00:31.082 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":21,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":21,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-11T22:59:59.460410+0000","modified":"2024-03-11T23:00:18.156048+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":47,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":2,"in":[0,1],"up":{"mds_0":14444,"mds_1":14460},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14444":{"gid":14444,"name":"cephfs.smithi153.qxjyxz","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.153:6827/633795058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":633795058},{"type":"v1","addr":"172.21.15.153:6827","nonce":633795058}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14460":{"gid":14460,"name":"cephfs.smithi153.yuzdaj","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.153:6829/2364288180","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":2364288180},{"type":"v1","addr":"172.21.15.153:6829","nonce":2364288180}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24263":{"gid":24263,"name":"cephfs.smithi163.kzggfd","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.163:6827/1010460391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":1010460391},{"type":"v1","addr":"172.21.15.163:6827","nonce":1010460391}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24267":{"gid":24267,"name":"cephfs.smithi163.pmppga","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6825/685283199","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":685283199},{"type":"v1","addr":"172.21.15.163:6825","nonce":685283199}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2024-03-11T23:00:31.082 INFO:teuthology.orchestra.run.smithi153.stderr:dumped fsmap epoch 21 2024-03-11T23:00:31.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:31.079+0000 7f52a2c6e640 1 -- 172.21.15.153:0/2944395128 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f5284069210 msgr2=0x7f528406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:31.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:31.079+0000 7f52a2c6e640 1 --2- 172.21.15.153:0/2944395128 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f5284069210 0x7f528406b6c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f529c10e080 tx=0x7f5290009da0 comp rx=0 tx=0).stop 2024-03-11T23:00:31.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:31.079+0000 7f52a2c6e640 1 -- 172.21.15.153:0/2944395128 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f529c0fc2a0 msgr2=0x7f529c10cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:31.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:31.079+0000 7f52a2c6e640 1 --2- 172.21.15.153:0/2944395128 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f529c0fc2a0 0x7f529c10cb30 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f528c002840 tx=0x7f528c004570 comp rx=0 tx=0).stop 2024-03-11T23:00:31.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:31.079+0000 7f52a2c6e640 1 -- 172.21.15.153:0/2944395128 shutdown_connections 2024-03-11T23:00:31.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:31.079+0000 7f52a2c6e640 1 --2- 172.21.15.153:0/2944395128 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f5284069210 0x7f528406b6c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:31.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:31.079+0000 7f52a2c6e640 1 --2- 172.21.15.153:0/2944395128 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f529c0fd4d0 0x7f529c10d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:31.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:31.079+0000 7f52a2c6e640 1 --2- 172.21.15.153:0/2944395128 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f529c0fc2a0 0x7f529c10cb30 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:31.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:31.079+0000 7f52a2c6e640 1 -- 172.21.15.153:0/2944395128 >> 172.21.15.153:0/2944395128 conn(0x7f529c0f7980 msgr2=0x7f529c100700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:00:31.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:31.079+0000 7f52a2c6e640 1 -- 172.21.15.153:0/2944395128 shutdown_connections 2024-03-11T23:00:31.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:31.079+0000 7f52a2c6e640 1 -- 172.21.15.153:0/2944395128 wait complete. 2024-03-11T23:00:31.094 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 21, 'max_mds': 2, 'flags': 50} 2024-03-11T23:00:31.094 INFO:teuthology.run_tasks:Running task kclient... 2024-03-11T23:00:31.122 INFO:tasks.kclient:Mounting kernel clients... 2024-03-11T23:00:31.122 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2024-03-11T23:00:31.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:00:31.122 DEBUG:teuthology.orchestra.run.smithi153:> ip netns list 2024-03-11T23:00:31.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:00:31.138 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link delete ceph-brx 2024-03-11T23:00:31.207 INFO:teuthology.orchestra.run.smithi153.stderr:Cannot find device "ceph-brx" 2024-03-11T23:00:31.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T23:00:31.209 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:00:31.210 DEBUG:teuthology.orchestra.run.smithi163:> ip netns list 2024-03-11T23:00:31.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:00:31.230 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip link delete ceph-brx 2024-03-11T23:00:31.307 INFO:teuthology.orchestra.run.smithi163.stderr:Cannot find device "ceph-brx" 2024-03-11T23:00:31.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T23:00:31.309 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-03-11T23:00:31.309 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-11T23:00:31.309 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-11T23:00:31.309 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph fs ls 2024-03-11T23:00:31.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:31 smithi153 ceph-mon[28962]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.7 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-11T23:00:31.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:31 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/3930979411' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-11T23:00:31.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:31 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/172910397' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-11T23:00:31.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:31 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/2944395128' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T23:00:31.569 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T23:00:31.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:31 smithi163 ceph-mon[35584]: pgmap v191: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 5.7 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-11T23:00:31.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:31 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/3930979411' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-11T23:00:31.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:31 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/172910397' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-11T23:00:31.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:31 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/2944395128' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-11T23:00:33.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.134+0000 7f586169f700 1 -- 172.21.15.153:0/4094023503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f585c1038f0 msgr2=0x7f585c0fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:33.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.134+0000 7f586169f700 1 --2- 172.21.15.153:0/4094023503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f585c1038f0 0x7f585c0fcf60 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f5850004940 tx=0x7f5850009e30 comp rx=0 tx=0).stop 2024-03-11T23:00:33.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.134+0000 7f586169f700 1 -- 172.21.15.153:0/4094023503 shutdown_connections 2024-03-11T23:00:33.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.134+0000 7f586169f700 1 --2- 172.21.15.153:0/4094023503 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f585c1038f0 0x7f585c0fcf60 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:33.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.134+0000 7f586169f700 1 --2- 172.21.15.153:0/4094023503 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f585c102f50 0x7f585c103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:33.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.134+0000 7f586169f700 1 -- 172.21.15.153:0/4094023503 >> 172.21.15.153:0/4094023503 conn(0x7f585c0f8ae0 msgr2=0x7f585c0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:00:33.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.134+0000 7f586169f700 1 -- 172.21.15.153:0/4094023503 shutdown_connections 2024-03-11T23:00:33.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.134+0000 7f586169f700 1 -- 172.21.15.153:0/4094023503 wait complete. 2024-03-11T23:00:33.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.135+0000 7f586169f700 1 Processor -- start 2024-03-11T23:00:33.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.135+0000 7f586169f700 1 -- start start 2024-03-11T23:00:33.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.135+0000 7f586169f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f585c102f50 0x7f585c113a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:33.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.135+0000 7f586169f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f585c1038f0 0x7f585c113fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:33.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.135+0000 7f586169f700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f585c114620 con 0x7f585c1038f0 2024-03-11T23:00:33.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.135+0000 7f586169f700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f585c10d950 con 0x7f585c102f50 2024-03-11T23:00:33.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.135+0000 7f585a7fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f585c1038f0 0x7f585c113fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:33.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.135+0000 7f585a7fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f585c1038f0 0x7f585c113fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:58026/0 (socket says 172.21.15.153:58026) 2024-03-11T23:00:33.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.135+0000 7f585a7fc700 1 -- 172.21.15.153:0/1651664741 learned_addr learned my addr 172.21.15.153:0/1651664741 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T23:00:33.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.135+0000 7f585affd700 1 --2- 172.21.15.153:0/1651664741 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f585c102f50 0x7f585c113a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:33.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.136+0000 7f585a7fc700 1 -- 172.21.15.153:0/1651664741 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f585c102f50 msgr2=0x7f585c113a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:33.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.136+0000 7f585a7fc700 1 --2- 172.21.15.153:0/1651664741 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f585c102f50 0x7f585c113a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:33.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.136+0000 7f585a7fc700 1 -- 172.21.15.153:0/1651664741 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5850005040 con 0x7f585c1038f0 2024-03-11T23:00:33.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.136+0000 7f585a7fc700 1 --2- 172.21.15.153:0/1651664741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f585c1038f0 0x7f585c113fd0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5850009150 tx=0x7f5850009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:00:33.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.136+0000 7f58437fe700 1 -- 172.21.15.153:0/1651664741 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f585000e040 con 0x7f585c1038f0 2024-03-11T23:00:33.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.136+0000 7f58437fe700 1 -- 172.21.15.153:0/1651664741 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f585000bbf0 con 0x7f585c1038f0 2024-03-11T23:00:33.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.136+0000 7f58437fe700 1 -- 172.21.15.153:0/1651664741 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f585001c630 con 0x7f585c1038f0 2024-03-11T23:00:33.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.136+0000 7f586169f700 1 -- 172.21.15.153:0/1651664741 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f585c10dbd0 con 0x7f585c1038f0 2024-03-11T23:00:33.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.136+0000 7f586169f700 1 -- 172.21.15.153:0/1651664741 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f585c10e110 con 0x7f585c1038f0 2024-03-11T23:00:33.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.137+0000 7f586169f700 1 -- 172.21.15.153:0/1651664741 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f585c105420 con 0x7f585c1038f0 2024-03-11T23:00:33.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.137+0000 7f58437fe700 1 -- 172.21.15.153:0/1651664741 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f585000bd60 con 0x7f585c1038f0 2024-03-11T23:00:33.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.138+0000 7f58437fe700 1 --2- 172.21.15.153:0/1651664741 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f58440692a0 0x7f584406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:33.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.138+0000 7f58437fe700 1 -- 172.21.15.153:0/1651664741 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f5850088600 con 0x7f585c1038f0 2024-03-11T23:00:33.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.138+0000 7f585affd700 1 --2- 172.21.15.153:0/1651664741 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f58440692a0 0x7f584406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:33.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.138+0000 7f585affd700 1 --2- 172.21.15.153:0/1651664741 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f58440692a0 0x7f584406b750 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f584c007fd0 tx=0x7f584c009040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:00:33.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.142+0000 7f58437fe700 1 -- 172.21.15.153:0/1651664741 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f5850058370 con 0x7f585c1038f0 2024-03-11T23:00:33.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.370+0000 7f586169f700 1 -- 172.21.15.153:0/1651664741 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f585c10a030 con 0x7f585c1038f0 2024-03-11T23:00:33.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.370+0000 7f58437fe700 1 -- 172.21.15.153:0/1651664741 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v21) v1 ==== 53+0+83 (secure 0 0 0) 0x7f5850057f00 con 0x7f585c1038f0 2024-03-11T23:00:33.375 INFO:teuthology.orchestra.run.smithi153.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-03-11T23:00:33.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.373+0000 7f586169f700 1 -- 172.21.15.153:0/1651664741 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f58440692a0 msgr2=0x7f584406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:33.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.373+0000 7f586169f700 1 --2- 172.21.15.153:0/1651664741 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f58440692a0 0x7f584406b750 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f584c007fd0 tx=0x7f584c009040 comp rx=0 tx=0).stop 2024-03-11T23:00:33.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.373+0000 7f586169f700 1 -- 172.21.15.153:0/1651664741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f585c1038f0 msgr2=0x7f585c113fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:33.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.373+0000 7f586169f700 1 --2- 172.21.15.153:0/1651664741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f585c1038f0 0x7f585c113fd0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5850009150 tx=0x7f5850009e30 comp rx=0 tx=0).stop 2024-03-11T23:00:33.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.373+0000 7f586169f700 1 -- 172.21.15.153:0/1651664741 shutdown_connections 2024-03-11T23:00:33.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.373+0000 7f586169f700 1 --2- 172.21.15.153:0/1651664741 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f58440692a0 0x7f584406b750 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:33.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.373+0000 7f586169f700 1 --2- 172.21.15.153:0/1651664741 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f585c102f50 0x7f585c113a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:33.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.373+0000 7f586169f700 1 --2- 172.21.15.153:0/1651664741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f585c1038f0 0x7f585c113fd0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:33.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.373+0000 7f586169f700 1 -- 172.21.15.153:0/1651664741 >> 172.21.15.153:0/1651664741 conn(0x7f585c0f8ae0 msgr2=0x7f585c0fa5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:00:33.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.373+0000 7f586169f700 1 -- 172.21.15.153:0/1651664741 shutdown_connections 2024-03-11T23:00:33.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:33.373+0000 7f586169f700 1 -- 172.21.15.153:0/1651664741 wait complete. 2024-03-11T23:00:33.691 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:33 smithi153 ceph-mon[28962]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-11T23:00:33.692 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:33 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:33.692 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:33 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:00:33.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:33 smithi163 ceph-mon[35584]: pgmap v192: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.3 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-11T23:00:33.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:33 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:00:33.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:33 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:00:34.000 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-11T23:00:34.000 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-11T23:00:34.000 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi153.front.sepia.ceph.com 2024-03-11T23:00:34.000 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-11T23:00:34.000 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-11T23:00:34.000 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-11T23:00:34.000 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-11T23:00:34.000 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-11T23:00:34.000 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-11T23:00:34.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:00:34.001 DEBUG:teuthology.orchestra.run.smithi153:> ip addr 2024-03-11T23:00:34.017 INFO:teuthology.orchestra.run.smithi153.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-11T23:00:34.017 INFO:teuthology.orchestra.run.smithi153.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-11T23:00:34.017 INFO:teuthology.orchestra.run.smithi153.stdout: inet 127.0.0.1/8 scope host lo 2024-03-11T23:00:34.017 INFO:teuthology.orchestra.run.smithi153.stdout: valid_lft forever preferred_lft forever 2024-03-11T23:00:34.018 INFO:teuthology.orchestra.run.smithi153.stdout: inet6 ::1/128 scope host 2024-03-11T23:00:34.018 INFO:teuthology.orchestra.run.smithi153.stdout: valid_lft forever preferred_lft forever 2024-03-11T23:00:34.018 INFO:teuthology.orchestra.run.smithi153.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-11T23:00:34.018 INFO:teuthology.orchestra.run.smithi153.stdout: link/ether 0c:c4:7a:da:6a:b2 brd ff:ff:ff:ff:ff:ff 2024-03-11T23:00:34.018 INFO:teuthology.orchestra.run.smithi153.stdout: altname enp7s0f0 2024-03-11T23:00:34.018 INFO:teuthology.orchestra.run.smithi153.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-11T23:00:34.018 INFO:teuthology.orchestra.run.smithi153.stdout: link/ether 0c:c4:7a:da:6a:b3 brd ff:ff:ff:ff:ff:ff 2024-03-11T23:00:34.018 INFO:teuthology.orchestra.run.smithi153.stdout: altname enp7s0f1 2024-03-11T23:00:34.018 INFO:teuthology.orchestra.run.smithi153.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-11T23:00:34.018 INFO:teuthology.orchestra.run.smithi153.stdout: link/ether 0c:c4:7a:88:6d:ea brd ff:ff:ff:ff:ff:ff 2024-03-11T23:00:34.018 INFO:teuthology.orchestra.run.smithi153.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-11T23:00:34.018 INFO:teuthology.orchestra.run.smithi153.stdout: link/ether 0c:c4:7a:88:6d:eb brd ff:ff:ff:ff:ff:ff 2024-03-11T23:00:34.018 INFO:teuthology.orchestra.run.smithi153.stdout: inet 172.21.15.153/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-11T23:00:34.018 INFO:teuthology.orchestra.run.smithi153.stdout: valid_lft 42098sec preferred_lft 42098sec 2024-03-11T23:00:34.019 INFO:teuthology.orchestra.run.smithi153.stdout: inet6 fe80::ec4:7aff:fe88:6deb/64 scope link 2024-03-11T23:00:34.019 INFO:teuthology.orchestra.run.smithi153.stdout: valid_lft forever preferred_lft forever 2024-03-11T23:00:34.019 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-11T23:00:34.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:00:34.019 DEBUG:teuthology.orchestra.run.smithi153:> (cd / && exec bash -c ' 2024-03-11T23:00:34.019 DEBUG:teuthology.orchestra.run.smithi153:> set -e 2024-03-11T23:00:34.019 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link add name ceph-brx type bridge 2024-03-11T23:00:34.019 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip addr flush dev ceph-brx 2024-03-11T23:00:34.019 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link set ceph-brx up 2024-03-11T23:00:34.020 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-11T23:00:34.020 DEBUG:teuthology.orchestra.run.smithi153:> ') 2024-03-11T23:00:34.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:00:34.200 DEBUG:teuthology.orchestra.run.smithi153:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-11T23:00:34.226 INFO:teuthology.orchestra.run.smithi153.stdout:1 2024-03-11T23:00:34.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:00:34.228 DEBUG:teuthology.orchestra.run.smithi153:> route 2024-03-11T23:00:34.294 INFO:teuthology.orchestra.run.smithi153.stdout:Kernel IP routing table 2024-03-11T23:00:34.294 INFO:teuthology.orchestra.run.smithi153.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-11T23:00:34.294 INFO:teuthology.orchestra.run.smithi153.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-11T23:00:34.295 INFO:teuthology.orchestra.run.smithi153.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-11T23:00:34.295 INFO:teuthology.orchestra.run.smithi153.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-11T23:00:34.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:00:34.295 DEBUG:teuthology.orchestra.run.smithi153:> (cd / && exec bash -c ' 2024-03-11T23:00:34.295 DEBUG:teuthology.orchestra.run.smithi153:> set -e 2024-03-11T23:00:34.295 DEBUG:teuthology.orchestra.run.smithi153:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-11T23:00:34.295 DEBUG:teuthology.orchestra.run.smithi153:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-11T23:00:34.295 DEBUG:teuthology.orchestra.run.smithi153:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-11T23:00:34.296 DEBUG:teuthology.orchestra.run.smithi153:> ') 2024-03-11T23:00:34.500 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:34 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/1651664741' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-11T23:00:34.614 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:00:34.614 DEBUG:teuthology.orchestra.run.smithi153:> ip netns list 2024-03-11T23:00:34.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:00:34.632 DEBUG:teuthology.orchestra.run.smithi153:> ip netns list-id 2024-03-11T23:00:34.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:00:34.687 DEBUG:teuthology.orchestra.run.smithi153:> (cd / && exec bash -c ' 2024-03-11T23:00:34.687 DEBUG:teuthology.orchestra.run.smithi153:> set -e 2024-03-11T23:00:34.687 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-11T23:00:34.687 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-11T23:00:34.688 DEBUG:teuthology.orchestra.run.smithi153:> ') 2024-03-11T23:00:34.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:34 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/1651664741' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-11T23:00:34.770 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-11T23:00:34.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:00:34.770 DEBUG:teuthology.orchestra.run.smithi153:> (cd / && exec bash -c ' 2024-03-11T23:00:34.770 DEBUG:teuthology.orchestra.run.smithi153:> set -e 2024-03-11T23:00:34.770 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-11T23:00:34.770 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-11T23:00:34.770 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-11T23:00:34.770 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-11T23:00:34.771 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-11T23:00:34.771 DEBUG:teuthology.orchestra.run.smithi153:> ') 2024-03-11T23:00:34.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:00:34.910 DEBUG:teuthology.orchestra.run.smithi153:> (cd / && exec bash -c ' 2024-03-11T23:00:34.910 DEBUG:teuthology.orchestra.run.smithi153:> set -e 2024-03-11T23:00:34.910 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link set brx.0 up 2024-03-11T23:00:34.910 DEBUG:teuthology.orchestra.run.smithi153:> sudo ip link set dev brx.0 master ceph-brx 2024-03-11T23:00:34.910 DEBUG:teuthology.orchestra.run.smithi153:> ') 2024-03-11T23:00:34.951 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-11T23:00:34.951 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-11T23:00:35.007 INFO:teuthology.orchestra.run.smithi153.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-11T23:00:35.007 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-11T23:00:35.007 DEBUG:teuthology.orchestra.run.smithi153:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-11T23:00:35.062 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-11T23:00:35.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:00:35.062 DEBUG:teuthology.orchestra.run.smithi153:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-11T23:00:35.155 INFO:teuthology.orchestra.run.smithi153.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-11T23:00:35.155 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-11T23:00:35.155 INFO:teuthology.orchestra.run.smithi153.stdout:invalid new device string format 2024-03-11T23:00:35.263 INFO:teuthology.orchestra.run.smithi153.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-11T23:00:35.263 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-11T23:00:35.263 INFO:teuthology.orchestra.run.smithi153.stdout:invalid new device string format 2024-03-11T23:00:35.263 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: resolved to: "172.21.15.153:3300,172.21.15.163:3300" 2024-03-11T23:00:35.263 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-11T23:00:35.268 INFO:teuthology.orchestra.run.smithi153.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-11T23:00:35.268 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-11T23:00:35.268 INFO:teuthology.orchestra.run.smithi153.stdout:invalid new device string format 2024-03-11T23:00:35.269 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: resolved to: "172.21.15.153:3300,172.21.15.163:3300" 2024-03-11T23:00:35.269 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-11T23:00:35.269 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: resolved to: "172.21.15.153:6789,172.21.15.163:6789" 2024-03-11T23:00:35.269 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.153:6789,172.21.15.163:6789:/ 2024-03-11T23:00:35.269 INFO:teuthology.orchestra.run.smithi153.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=fa913b96-dff9-11ee-95c8-87774f69a715" will pass to kernel 2024-03-11T23:00:35.270 INFO:teuthology.orchestra.run.smithi153.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-11T23:00:35.270 INFO:teuthology.orchestra.run.smithi153.stdout: You just mounted a file system that supports labels which does not 2024-03-11T23:00:35.270 INFO:teuthology.orchestra.run.smithi153.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-11T23:00:35.270 INFO:teuthology.orchestra.run.smithi153.stdout: applications will generate AVC messages and not be allowed access to 2024-03-11T23:00:35.270 INFO:teuthology.orchestra.run.smithi153.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-11T23:00:35.271 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-11T23:00:35.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:00:35.272 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-11T23:00:35.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:00:35.305 DEBUG:teuthology.orchestra.run.smithi153:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-11T23:00:35.369 INFO:teuthology.orchestra.run.smithi153.stdout:client14508 2024-03-11T23:00:35.369 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-11T23:00:35.370 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-11T23:00:35.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:00:35.427 DEBUG:teuthology.orchestra.run.smithi153:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-11T23:00:35.489 INFO:teuthology.orchestra.run.smithi153.stdout:client14508 2024-03-11T23:00:35.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:00:35.489 DEBUG:teuthology.orchestra.run.smithi153:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-11T23:00:35.548 INFO:teuthology.orchestra.run.smithi153.stdout:client14508 2024-03-11T23:00:35.548 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:00:35.549 DEBUG:teuthology.orchestra.run.smithi153:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/fa913b96-dff9-11ee-95c8-87774f69a715.client14508/status') 2024-03-11T23:00:35.617 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:35 smithi153 ceph-mon[28962]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-11T23:00:35.619 INFO:teuthology.orchestra.run.smithi153.stdout:instance: client.14508 (3)192.168.0.1:0/3469210821 2024-03-11T23:00:35.619 INFO:teuthology.orchestra.run.smithi153.stdout:blocklisted: false 2024-03-11T23:00:35.620 INFO:teuthology.orchestra.run.smithi153.stderr:0+1 records in 2024-03-11T23:00:35.620 INFO:teuthology.orchestra.run.smithi153.stderr:0+1 records out 2024-03-11T23:00:35.620 INFO:teuthology.orchestra.run.smithi153.stderr:70 bytes copied, 2.0996e-05 s, 3.3 MB/s 2024-03-11T23:00:35.621 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-03-11T23:00:35.621 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-11T23:00:35.621 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-11T23:00:35.621 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -- ceph fs ls 2024-03-11T23:00:35.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:35 smithi163 ceph-mon[35584]: pgmap v193: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-11T23:00:35.928 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T23:00:37.613 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:37 smithi153 ceph-mon[28962]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-11T23:00:37.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.624+0000 7f2225894700 1 -- 172.21.15.153:0/1242357223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22200fdfe0 msgr2=0x7f22200fe3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:37.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.624+0000 7f2225894700 1 --2- 172.21.15.153:0/1242357223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22200fdfe0 0x7f22200fe3b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2218004970 tx=0x7f2218009e30 comp rx=0 tx=0).stop 2024-03-11T23:00:37.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.624+0000 7f2225894700 1 -- 172.21.15.153:0/1242357223 shutdown_connections 2024-03-11T23:00:37.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.624+0000 7f2225894700 1 --2- 172.21.15.153:0/1242357223 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22200fe8f0 0x7f2220107050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:37.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.624+0000 7f2225894700 1 --2- 172.21.15.153:0/1242357223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22200fdfe0 0x7f22200fe3b0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:37.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.624+0000 7f2225894700 1 -- 172.21.15.153:0/1242357223 >> 172.21.15.153:0/1242357223 conn(0x7f2220071d90 msgr2=0x7f2220072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:00:37.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.624+0000 7f2225894700 1 -- 172.21.15.153:0/1242357223 shutdown_connections 2024-03-11T23:00:37.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.624+0000 7f2225894700 1 -- 172.21.15.153:0/1242357223 wait complete. 2024-03-11T23:00:37.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.625+0000 7f2225894700 1 Processor -- start 2024-03-11T23:00:37.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.625+0000 7f2225894700 1 -- start start 2024-03-11T23:00:37.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.625+0000 7f2225894700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22200fdfe0 0x7f22200fbea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:37.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.625+0000 7f2225894700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22200fe8f0 0x7f22200fc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:37.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.625+0000 7f2225894700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22200fc9e0 con 0x7f22200fdfe0 2024-03-11T23:00:37.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.625+0000 7f2225894700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22200fcb50 con 0x7f22200fe8f0 2024-03-11T23:00:37.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.625+0000 7f221effd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22200fdfe0 0x7f22200fbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:37.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.625+0000 7f221effd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22200fdfe0 0x7f22200fbea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:58054/0 (socket says 172.21.15.153:58054) 2024-03-11T23:00:37.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.625+0000 7f221effd700 1 -- 172.21.15.153:0/22841544 learned_addr learned my addr 172.21.15.153:0/22841544 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T23:00:37.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.625+0000 7f221e7fc700 1 --2- 172.21.15.153:0/22841544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22200fe8f0 0x7f22200fc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:37.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.625+0000 7f221effd700 1 -- 172.21.15.153:0/22841544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22200fe8f0 msgr2=0x7f22200fc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:37.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.625+0000 7f221effd700 1 --2- 172.21.15.153:0/22841544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22200fe8f0 0x7f22200fc3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:37.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.625+0000 7f221effd700 1 -- 172.21.15.153:0/22841544 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2218005040 con 0x7f22200fdfe0 2024-03-11T23:00:37.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.626+0000 7f221effd700 1 --2- 172.21.15.153:0/22841544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22200fdfe0 0x7f22200fbea0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2218004f50 tx=0x7f2218009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:00:37.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.626+0000 7f22077fe700 1 -- 172.21.15.153:0/22841544 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f221800e040 con 0x7f22200fdfe0 2024-03-11T23:00:37.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.626+0000 7f22077fe700 1 -- 172.21.15.153:0/22841544 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2218004290 con 0x7f22200fdfe0 2024-03-11T23:00:37.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.626+0000 7f2225894700 1 -- 172.21.15.153:0/22841544 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2220119c90 con 0x7f22200fdfe0 2024-03-11T23:00:37.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.626+0000 7f22077fe700 1 -- 172.21.15.153:0/22841544 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f221801c3f0 con 0x7f22200fdfe0 2024-03-11T23:00:37.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.626+0000 7f2225894700 1 -- 172.21.15.153:0/22841544 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f222011a0f0 con 0x7f22200fdfe0 2024-03-11T23:00:37.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.627+0000 7f2225894700 1 -- 172.21.15.153:0/22841544 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22201046c0 con 0x7f22200fdfe0 2024-03-11T23:00:37.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.627+0000 7f22077fe700 1 -- 172.21.15.153:0/22841544 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f221800bc10 con 0x7f22200fdfe0 2024-03-11T23:00:37.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.628+0000 7f22077fe700 1 --2- 172.21.15.153:0/22841544 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f22080692b0 0x7f220806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:37.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.628+0000 7f22077fe700 1 -- 172.21.15.153:0/22841544 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f2218088420 con 0x7f22200fdfe0 2024-03-11T23:00:37.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.628+0000 7f221e7fc700 1 --2- 172.21.15.153:0/22841544 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f22080692b0 0x7f220806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:37.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.628+0000 7f221e7fc700 1 --2- 172.21.15.153:0/22841544 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f22080692b0 0x7f220806b760 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f22200fd8a0 tx=0x7f2214009e20 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:00:37.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.632+0000 7f22077fe700 1 -- 172.21.15.153:0/22841544 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f2218054700 con 0x7f22200fdfe0 2024-03-11T23:00:37.737 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:37 smithi163 ceph-mon[35584]: pgmap v194: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-11T23:00:37.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.857+0000 7f2225894700 1 -- 172.21.15.153:0/22841544 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f222004bed0 con 0x7f22200fdfe0 2024-03-11T23:00:37.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.857+0000 7f22077fe700 1 -- 172.21.15.153:0/22841544 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v21) v1 ==== 53+0+83 (secure 0 0 0) 0x7f2218057d20 con 0x7f22200fdfe0 2024-03-11T23:00:37.862 INFO:teuthology.orchestra.run.smithi153.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-03-11T23:00:37.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.859+0000 7f2225894700 1 -- 172.21.15.153:0/22841544 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f22080692b0 msgr2=0x7f220806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:37.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.859+0000 7f2225894700 1 --2- 172.21.15.153:0/22841544 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f22080692b0 0x7f220806b760 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f22200fd8a0 tx=0x7f2214009e20 comp rx=0 tx=0).stop 2024-03-11T23:00:37.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.859+0000 7f2225894700 1 -- 172.21.15.153:0/22841544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22200fdfe0 msgr2=0x7f22200fbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:37.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.859+0000 7f2225894700 1 --2- 172.21.15.153:0/22841544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22200fdfe0 0x7f22200fbea0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2218004f50 tx=0x7f2218009e30 comp rx=0 tx=0).stop 2024-03-11T23:00:37.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.859+0000 7f2225894700 1 -- 172.21.15.153:0/22841544 shutdown_connections 2024-03-11T23:00:37.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.859+0000 7f2225894700 1 --2- 172.21.15.153:0/22841544 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22200fdfe0 0x7f22200fbea0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:37.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.859+0000 7f2225894700 1 --2- 172.21.15.153:0/22841544 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f22080692b0 0x7f220806b760 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:37.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.859+0000 7f2225894700 1 --2- 172.21.15.153:0/22841544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22200fe8f0 0x7f22200fc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:37.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.859+0000 7f2225894700 1 -- 172.21.15.153:0/22841544 >> 172.21.15.153:0/22841544 conn(0x7f2220071d90 msgr2=0x7f2220101840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:00:37.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.860+0000 7f2225894700 1 -- 172.21.15.153:0/22841544 shutdown_connections 2024-03-11T23:00:37.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:37.860+0000 7f2225894700 1 -- 172.21.15.153:0/22841544 wait complete. 2024-03-11T23:00:38.458 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:38 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/22841544' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-11T23:00:38.481 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-11T23:00:38.481 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-11T23:00:38.481 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi163.front.sepia.ceph.com 2024-03-11T23:00:38.481 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-11T23:00:38.481 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-11T23:00:38.482 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-11T23:00:38.482 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-11T23:00:38.482 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-11T23:00:38.482 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-11T23:00:38.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:00:38.482 DEBUG:teuthology.orchestra.run.smithi163:> ip addr 2024-03-11T23:00:38.487 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:38 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/22841544' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-11T23:00:38.500 INFO:teuthology.orchestra.run.smithi163.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-11T23:00:38.500 INFO:teuthology.orchestra.run.smithi163.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-11T23:00:38.500 INFO:teuthology.orchestra.run.smithi163.stdout: inet 127.0.0.1/8 scope host lo 2024-03-11T23:00:38.500 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-03-11T23:00:38.501 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 ::1/128 scope host 2024-03-11T23:00:38.501 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-03-11T23:00:38.501 INFO:teuthology.orchestra.run.smithi163.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-11T23:00:38.501 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 0c:c4:7a:d9:38:06 brd ff:ff:ff:ff:ff:ff 2024-03-11T23:00:38.501 INFO:teuthology.orchestra.run.smithi163.stdout: altname enp7s0f0 2024-03-11T23:00:38.501 INFO:teuthology.orchestra.run.smithi163.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-11T23:00:38.501 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 0c:c4:7a:d9:38:07 brd ff:ff:ff:ff:ff:ff 2024-03-11T23:00:38.501 INFO:teuthology.orchestra.run.smithi163.stdout: altname enp7s0f1 2024-03-11T23:00:38.501 INFO:teuthology.orchestra.run.smithi163.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-11T23:00:38.501 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 0c:c4:7a:88:7d:5a brd ff:ff:ff:ff:ff:ff 2024-03-11T23:00:38.502 INFO:teuthology.orchestra.run.smithi163.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-11T23:00:38.502 INFO:teuthology.orchestra.run.smithi163.stdout: link/ether 0c:c4:7a:88:7d:5b brd ff:ff:ff:ff:ff:ff 2024-03-11T23:00:38.503 INFO:teuthology.orchestra.run.smithi163.stdout: inet 172.21.15.163/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-11T23:00:38.503 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft 42246sec preferred_lft 42246sec 2024-03-11T23:00:38.503 INFO:teuthology.orchestra.run.smithi163.stdout: inet6 fe80::ec4:7aff:fe88:7d5b/64 scope link 2024-03-11T23:00:38.503 INFO:teuthology.orchestra.run.smithi163.stdout: valid_lft forever preferred_lft forever 2024-03-11T23:00:38.503 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-11T23:00:38.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:00:38.503 DEBUG:teuthology.orchestra.run.smithi163:> (cd / && exec bash -c ' 2024-03-11T23:00:38.504 DEBUG:teuthology.orchestra.run.smithi163:> set -e 2024-03-11T23:00:38.504 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip link add name ceph-brx type bridge 2024-03-11T23:00:38.504 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip addr flush dev ceph-brx 2024-03-11T23:00:38.504 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip link set ceph-brx up 2024-03-11T23:00:38.504 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-11T23:00:38.504 DEBUG:teuthology.orchestra.run.smithi163:> ') 2024-03-11T23:00:38.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:00:38.688 DEBUG:teuthology.orchestra.run.smithi163:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-11T23:00:38.716 INFO:teuthology.orchestra.run.smithi163.stdout:1 2024-03-11T23:00:38.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:00:38.718 DEBUG:teuthology.orchestra.run.smithi163:> route 2024-03-11T23:00:38.789 INFO:teuthology.orchestra.run.smithi163.stdout:Kernel IP routing table 2024-03-11T23:00:38.789 INFO:teuthology.orchestra.run.smithi163.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-11T23:00:38.790 INFO:teuthology.orchestra.run.smithi163.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-11T23:00:38.790 INFO:teuthology.orchestra.run.smithi163.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-11T23:00:38.790 INFO:teuthology.orchestra.run.smithi163.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-11T23:00:38.790 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:00:38.791 DEBUG:teuthology.orchestra.run.smithi163:> (cd / && exec bash -c ' 2024-03-11T23:00:38.791 DEBUG:teuthology.orchestra.run.smithi163:> set -e 2024-03-11T23:00:38.791 DEBUG:teuthology.orchestra.run.smithi163:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-11T23:00:38.791 DEBUG:teuthology.orchestra.run.smithi163:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-11T23:00:38.791 DEBUG:teuthology.orchestra.run.smithi163:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-11T23:00:38.791 DEBUG:teuthology.orchestra.run.smithi163:> ') 2024-03-11T23:00:39.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:00:39.041 DEBUG:teuthology.orchestra.run.smithi163:> ip netns list 2024-03-11T23:00:39.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:00:39.056 DEBUG:teuthology.orchestra.run.smithi163:> ip netns list-id 2024-03-11T23:00:39.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:00:39.111 DEBUG:teuthology.orchestra.run.smithi163:> (cd / && exec bash -c ' 2024-03-11T23:00:39.112 DEBUG:teuthology.orchestra.run.smithi163:> set -e 2024-03-11T23:00:39.112 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-11T23:00:39.112 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-11T23:00:39.112 DEBUG:teuthology.orchestra.run.smithi163:> ') 2024-03-11T23:00:39.197 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-11T23:00:39.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:00:39.197 DEBUG:teuthology.orchestra.run.smithi163:> (cd / && exec bash -c ' 2024-03-11T23:00:39.197 DEBUG:teuthology.orchestra.run.smithi163:> set -e 2024-03-11T23:00:39.197 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-11T23:00:39.197 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-11T23:00:39.197 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-11T23:00:39.197 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-11T23:00:39.197 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-11T23:00:39.197 DEBUG:teuthology.orchestra.run.smithi163:> ') 2024-03-11T23:00:39.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:00:39.331 DEBUG:teuthology.orchestra.run.smithi163:> (cd / && exec bash -c ' 2024-03-11T23:00:39.332 DEBUG:teuthology.orchestra.run.smithi163:> set -e 2024-03-11T23:00:39.332 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip link set brx.0 up 2024-03-11T23:00:39.332 DEBUG:teuthology.orchestra.run.smithi163:> sudo ip link set dev brx.0 master ceph-brx 2024-03-11T23:00:39.332 DEBUG:teuthology.orchestra.run.smithi163:> ') 2024-03-11T23:00:39.374 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-11T23:00:39.374 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-11T23:00:39.429 INFO:teuthology.orchestra.run.smithi163.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-11T23:00:39.429 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-11T23:00:39.429 DEBUG:teuthology.orchestra.run.smithi163:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-11T23:00:39.485 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-11T23:00:39.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:00:39.486 DEBUG:teuthology.orchestra.run.smithi163:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-11T23:00:39.487 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:39 smithi163 ceph-mon[35584]: pgmap v195: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-11T23:00:39.539 INFO:teuthology.orchestra.run.smithi163.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-11T23:00:39.540 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-11T23:00:39.540 INFO:teuthology.orchestra.run.smithi163.stdout:invalid new device string format 2024-03-11T23:00:39.658 INFO:teuthology.orchestra.run.smithi163.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-11T23:00:39.658 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-11T23:00:39.658 INFO:teuthology.orchestra.run.smithi163.stdout:invalid new device string format 2024-03-11T23:00:39.658 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: resolved to: "172.21.15.153:3300,172.21.15.163:3300" 2024-03-11T23:00:39.659 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-11T23:00:39.664 INFO:teuthology.orchestra.run.smithi163.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-11T23:00:39.664 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-11T23:00:39.664 INFO:teuthology.orchestra.run.smithi163.stdout:invalid new device string format 2024-03-11T23:00:39.664 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: resolved to: "172.21.15.153:3300,172.21.15.163:3300" 2024-03-11T23:00:39.664 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-11T23:00:39.664 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: resolved to: "172.21.15.153:6789,172.21.15.163:6789" 2024-03-11T23:00:39.664 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.153:6789,172.21.15.163:6789:/ 2024-03-11T23:00:39.664 INFO:teuthology.orchestra.run.smithi163.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=fa913b96-dff9-11ee-95c8-87774f69a715" will pass to kernel 2024-03-11T23:00:39.665 INFO:teuthology.orchestra.run.smithi163.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-03-11T23:00:39.665 INFO:teuthology.orchestra.run.smithi163.stdout: You just mounted a file system that supports labels which does not 2024-03-11T23:00:39.665 INFO:teuthology.orchestra.run.smithi163.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-11T23:00:39.665 INFO:teuthology.orchestra.run.smithi163.stdout: applications will generate AVC messages and not be allowed access to 2024-03-11T23:00:39.665 INFO:teuthology.orchestra.run.smithi163.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-11T23:00:39.666 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-11T23:00:39.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:00:39.666 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-11T23:00:39.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:00:39.695 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-11T23:00:39.752 INFO:teuthology.orchestra.run.smithi163.stdout:client14514 2024-03-11T23:00:39.752 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-11T23:00:39.752 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-11T23:00:39.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:39 smithi153 ceph-mon[28962]: pgmap v195: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-11T23:00:39.809 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:00:39.809 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-11T23:00:39.866 INFO:teuthology.orchestra.run.smithi163.stdout:client14514 2024-03-11T23:00:39.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:00:39.866 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-11T23:00:39.922 INFO:teuthology.orchestra.run.smithi163.stdout:client14514 2024-03-11T23:00:39.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:00:39.923 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/fa913b96-dff9-11ee-95c8-87774f69a715.client14514/status') 2024-03-11T23:00:39.989 INFO:teuthology.orchestra.run.smithi163.stdout:instance: client.14514 (3)172.21.15.163:0/3672743676 2024-03-11T23:00:39.989 INFO:teuthology.orchestra.run.smithi163.stdout:blocklisted: false 2024-03-11T23:00:39.989 INFO:teuthology.orchestra.run.smithi163.stderr:0+1 records in 2024-03-11T23:00:39.989 INFO:teuthology.orchestra.run.smithi163.stderr:0+1 records out 2024-03-11T23:00:39.989 INFO:teuthology.orchestra.run.smithi163.stderr:72 bytes copied, 2.015e-05 s, 3.6 MB/s 2024-03-11T23:00:39.990 INFO:teuthology.run_tasks:Running task print... 2024-03-11T23:00:40.001 INFO:teuthology.task.print:**** done client 2024-03-11T23:00:40.001 INFO:teuthology.run_tasks:Running task parallel... 2024-03-11T23:00:40.015 INFO:teuthology.task.parallel:starting parallel... 2024-03-11T23:00:40.015 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-03-11T23:00:40.016 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-11T23:00:40.016 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi153.front.sepia.ceph.com 2024-03-11T23:00:40.016 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-03-11T23:00:40.017 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-03-11T23:00:40.017 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-03-11T23:00:40.020 INFO:tasks.workunit:Pulling workunits from ref fbcdfe674e56d1ea7a0810744e5915bb07961be5 2024-03-11T23:00:40.021 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-11T23:00:40.021 INFO:tasks.workunit:timeout=3h 2024-03-11T23:00:40.021 INFO:tasks.workunit:cleanup=True 2024-03-11T23:00:40.021 DEBUG:teuthology.orchestra.run.smithi153:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-11T23:00:40.041 INFO:teuthology.orchestra.run.smithi153.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-11T23:00:40.041 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-11T23:00:40.041 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 94h/148d Inode: 1 Links: 2 2024-03-11T23:00:40.041 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T23:00:40.041 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-11T23:00:40.041 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-03-11T23:00:40.041 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-11 23:00:04.114375551 +0000 2024-03-11T23:00:40.042 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-11 23:00:35.298120417 +0000 2024-03-11T23:00:40.042 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-11 23:00:04.114375551 +0000 2024-03-11T23:00:40.042 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-11T23:00:40.042 DEBUG:teuthology.orchestra.run.smithi153:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-11T23:00:40.115 DEBUG:teuthology.orchestra.run.smithi163:> stat -- /home/ubuntu/cephtest/mnt.1 2024-03-11T23:00:40.132 INFO:teuthology.orchestra.run.smithi163.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-03-11T23:00:40.132 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 1 Blocks: 0 IO Block: 65536 directory 2024-03-11T23:00:40.132 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 73h/115d Inode: 1 Links: 3 2024-03-11T23:00:40.132 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T23:00:40.132 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-11T23:00:40.132 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-03-11T23:00:40.132 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-03-11 23:00:40.106035716 +0000 2024-03-11T23:00:40.132 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-03-11 23:00:40.106035716 +0000 2024-03-11T23:00:40.132 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-03-11 23:00:04.114375551 +0000 2024-03-11T23:00:40.133 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-03-11T23:00:40.133 DEBUG:teuthology.orchestra.run.smithi163:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-03-11T23:00:40.206 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout fbcdfe674e56d1ea7a0810744e5915bb07961be5 2024-03-11T23:00:40.207 DEBUG:teuthology.orchestra.run.smithi163:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout fbcdfe674e56d1ea7a0810744e5915bb07961be5 2024-03-11T23:00:40.222 INFO:tasks.workunit.client.0.smithi153.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-11T23:00:40.263 INFO:tasks.workunit.client.1.smithi163.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-03-11T23:00:40.296 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T23:00:41.701 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:41 smithi153 ceph-mon[28962]: pgmap v196: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 255 B/s wr, 4 op/s 2024-03-11T23:00:41.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:41 smithi163 ceph-mon[35584]: pgmap v196: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 255 B/s wr, 4 op/s 2024-03-11T23:00:41.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:41.928+0000 7fa2788d8700 1 -- 172.21.15.153:0/1638874066 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2741000a0 msgr2=0x7fa274102520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:41.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:41.928+0000 7fa2788d8700 1 --2- 172.21.15.153:0/1638874066 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2741000a0 0x7fa274102520 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa268004880 tx=0x7fa268009e30 comp rx=0 tx=0).stop 2024-03-11T23:00:41.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:41.928+0000 7fa2788d8700 1 -- 172.21.15.153:0/1638874066 shutdown_connections 2024-03-11T23:00:41.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:41.928+0000 7fa2788d8700 1 --2- 172.21.15.153:0/1638874066 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa2741000a0 0x7fa274102520 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:41.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:41.928+0000 7fa2788d8700 1 --2- 172.21.15.153:0/1638874066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa274066c90 0x7fa2740ffad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:41.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:41.928+0000 7fa2788d8700 1 -- 172.21.15.153:0/1638874066 >> 172.21.15.153:0/1638874066 conn(0x7fa2740f7520 msgr2=0x7fa2740f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:00:41.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:41.928+0000 7fa2788d8700 1 -- 172.21.15.153:0/1638874066 shutdown_connections 2024-03-11T23:00:41.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:41.928+0000 7fa2788d8700 1 -- 172.21.15.153:0/1638874066 wait complete. 2024-03-11T23:00:41.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:41.929+0000 7fa2788d8700 1 Processor -- start 2024-03-11T23:00:41.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:41.929+0000 7fa2788d8700 1 -- start start 2024-03-11T23:00:41.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:41.929+0000 7fa2788d8700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa274066c90 0x7fa27410d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:41.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:41.929+0000 7fa2788d8700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa2741000a0 0x7fa27410de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:41.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:41.929+0000 7fa2788d8700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27410e480 con 0x7fa274066c90 2024-03-11T23:00:41.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:41.929+0000 7fa2788d8700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa27410ed20 con 0x7fa2741000a0 2024-03-11T23:00:41.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:41.929+0000 7fa27259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa274066c90 0x7fa27410d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:41.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:41.929+0000 7fa27259c700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa274066c90 0x7fa27410d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:32808/0 (socket says 172.21.15.153:32808) 2024-03-11T23:00:41.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:41.929+0000 7fa27259c700 1 -- 172.21.15.153:0/4126194041 learned_addr learned my addr 172.21.15.153:0/4126194041 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T23:00:41.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:41.929+0000 7fa271d9b700 1 --2- 172.21.15.153:0/4126194041 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa2741000a0 0x7fa27410de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:41.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:41.930+0000 7fa27259c700 1 -- 172.21.15.153:0/4126194041 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa2741000a0 msgr2=0x7fa27410de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:41.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:41.930+0000 7fa27259c700 1 --2- 172.21.15.153:0/4126194041 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa2741000a0 0x7fa27410de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:41.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:41.930+0000 7fa27259c700 1 -- 172.21.15.153:0/4126194041 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa268005040 con 0x7fa274066c90 2024-03-11T23:00:41.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:41.930+0000 7fa27259c700 1 --2- 172.21.15.153:0/4126194041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa274066c90 0x7fa27410d940 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa2640041d0 tx=0x7fa26400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:00:41.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:41.930+0000 7fa262ffd700 1 -- 172.21.15.153:0/4126194041 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa264007cc0 con 0x7fa274066c90 2024-03-11T23:00:41.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:41.930+0000 7fa262ffd700 1 -- 172.21.15.153:0/4126194041 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa26400f040 con 0x7fa274066c90 2024-03-11T23:00:41.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:41.930+0000 7fa2788d8700 1 -- 172.21.15.153:0/4126194041 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa27410efa0 con 0x7fa274066c90 2024-03-11T23:00:41.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:41.930+0000 7fa262ffd700 1 -- 172.21.15.153:0/4126194041 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa26400ca50 con 0x7fa274066c90 2024-03-11T23:00:41.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:41.930+0000 7fa2788d8700 1 -- 172.21.15.153:0/4126194041 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa27410f4e0 con 0x7fa274066c90 2024-03-11T23:00:41.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:41.931+0000 7fa2788d8700 1 -- 172.21.15.153:0/4126194041 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa274107a50 con 0x7fa274066c90 2024-03-11T23:00:41.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:41.931+0000 7fa262ffd700 1 -- 172.21.15.153:0/4126194041 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fa26400ccc0 con 0x7fa274066c90 2024-03-11T23:00:41.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:41.932+0000 7fa262ffd700 1 --2- 172.21.15.153:0/4126194041 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fa25c0692a0 0x7fa25c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:41.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:41.932+0000 7fa262ffd700 1 -- 172.21.15.153:0/4126194041 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fa264086c40 con 0x7fa274066c90 2024-03-11T23:00:41.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:41.932+0000 7fa271d9b700 1 --2- 172.21.15.153:0/4126194041 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fa25c0692a0 0x7fa25c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:41.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:41.932+0000 7fa271d9b700 1 --2- 172.21.15.153:0/4126194041 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fa25c0692a0 0x7fa25c06b750 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa2680137c0 tx=0x7fa268009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:00:41.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:41.936+0000 7fa262ffd700 1 -- 172.21.15.153:0/4126194041 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7fa26404dd20 con 0x7fa274066c90 2024-03-11T23:00:42.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:42.125+0000 7fa2788d8700 1 -- 172.21.15.153:0/4126194041 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7fa27404bed0 con 0x7fa274066c90 2024-03-11T23:00:42.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:42.125+0000 7fa262ffd700 1 -- 172.21.15.153:0/4126194041 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7fa2640569e0 con 0x7fa274066c90 2024-03-11T23:00:42.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:42.127+0000 7fa2788d8700 1 -- 172.21.15.153:0/4126194041 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fa25c0692a0 msgr2=0x7fa25c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:42.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:42.127+0000 7fa2788d8700 1 --2- 172.21.15.153:0/4126194041 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fa25c0692a0 0x7fa25c06b750 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa2680137c0 tx=0x7fa268009e30 comp rx=0 tx=0).stop 2024-03-11T23:00:42.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:42.127+0000 7fa2788d8700 1 -- 172.21.15.153:0/4126194041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa274066c90 msgr2=0x7fa27410d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:42.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:42.127+0000 7fa2788d8700 1 --2- 172.21.15.153:0/4126194041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa274066c90 0x7fa27410d940 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa2640041d0 tx=0x7fa26400b040 comp rx=0 tx=0).stop 2024-03-11T23:00:42.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:42.128+0000 7fa2788d8700 1 -- 172.21.15.153:0/4126194041 shutdown_connections 2024-03-11T23:00:42.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:42.128+0000 7fa2788d8700 1 --2- 172.21.15.153:0/4126194041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa274066c90 0x7fa27410d940 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:42.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:42.128+0000 7fa2788d8700 1 --2- 172.21.15.153:0/4126194041 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fa25c0692a0 0x7fa25c06b750 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:42.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:42.128+0000 7fa2788d8700 1 --2- 172.21.15.153:0/4126194041 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa2741000a0 0x7fa27410de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:42.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:42.128+0000 7fa2788d8700 1 -- 172.21.15.153:0/4126194041 >> 172.21.15.153:0/4126194041 conn(0x7fa2740f7520 msgr2=0x7fa2740f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:00:42.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:42.128+0000 7fa2788d8700 1 -- 172.21.15.153:0/4126194041 shutdown_connections 2024-03-11T23:00:42.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:42.128+0000 7fa2788d8700 1 -- 172.21.15.153:0/4126194041 wait complete. 2024-03-11T23:00:42.729 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-03-11T23:00:43.033 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T23:00:43.408 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:43 smithi153 ceph-mon[28962]: pgmap v197: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 341 B/s wr, 4 op/s 2024-03-11T23:00:43.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:43 smithi163 ceph-mon[35584]: pgmap v197: 65 pgs: 65 active+clean; 581 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 341 B/s wr, 4 op/s 2024-03-11T23:00:44.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.690+0000 7f0cb3ffc700 1 -- 172.21.15.153:0/2139689022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cac0fbe80 msgr2=0x7f0cac0fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:44.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.690+0000 7f0cb3ffc700 1 --2- 172.21.15.153:0/2139689022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cac0fbe80 0x7f0cac0fc290 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8004940 tx=0x7f0ca8009e30 comp rx=0 tx=0).stop 2024-03-11T23:00:44.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.690+0000 7f0cb3ffc700 1 -- 172.21.15.153:0/2139689022 shutdown_connections 2024-03-11T23:00:44.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.690+0000 7f0cb3ffc700 1 --2- 172.21.15.153:0/2139689022 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0cac0fd080 0x7f0cac0fd4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:44.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.690+0000 7f0cb3ffc700 1 --2- 172.21.15.153:0/2139689022 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cac0fbe80 0x7f0cac0fc290 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:44.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.690+0000 7f0cb3ffc700 1 -- 172.21.15.153:0/2139689022 >> 172.21.15.153:0/2139689022 conn(0x7f0cac0f7590 msgr2=0x7f0cac0f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:00:44.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.690+0000 7f0cb3ffc700 1 -- 172.21.15.153:0/2139689022 shutdown_connections 2024-03-11T23:00:44.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.690+0000 7f0cb3ffc700 1 -- 172.21.15.153:0/2139689022 wait complete. 2024-03-11T23:00:44.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.691+0000 7f0cb3ffc700 1 Processor -- start 2024-03-11T23:00:44.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.691+0000 7f0cb3ffc700 1 -- start start 2024-03-11T23:00:44.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.691+0000 7f0cb3ffc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cac0fbe80 0x7f0cac112ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:44.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.691+0000 7f0cb3ffc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0cac0fd080 0x7f0cac115000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:44.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.691+0000 7f0cb3ffc700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cac066c00 con 0x7f0cac0fbe80 2024-03-11T23:00:44.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.691+0000 7f0cb3ffc700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cac066d40 con 0x7f0cac0fd080 2024-03-11T23:00:44.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.691+0000 7f0cb1d98700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cac0fbe80 0x7f0cac112ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:44.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.691+0000 7f0cb1d98700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cac0fbe80 0x7f0cac112ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:32828/0 (socket says 172.21.15.153:32828) 2024-03-11T23:00:44.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.691+0000 7f0cb1d98700 1 -- 172.21.15.153:0/616351500 learned_addr learned my addr 172.21.15.153:0/616351500 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T23:00:44.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.691+0000 7f0cb1597700 1 --2- 172.21.15.153:0/616351500 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0cac0fd080 0x7f0cac115000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:44.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.691+0000 7f0cb1d98700 1 -- 172.21.15.153:0/616351500 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0cac0fd080 msgr2=0x7f0cac115000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:44.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.692+0000 7f0cb1d98700 1 --2- 172.21.15.153:0/616351500 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0cac0fd080 0x7f0cac115000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:44.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.692+0000 7f0cb1d98700 1 -- 172.21.15.153:0/616351500 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca8005040 con 0x7f0cac0fbe80 2024-03-11T23:00:44.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.692+0000 7f0cb1d98700 1 --2- 172.21.15.153:0/616351500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cac0fbe80 0x7f0cac112ab0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0ca800b3b0 tx=0x7f0ca8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:00:44.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.692+0000 7f0ca27fc700 1 -- 172.21.15.153:0/616351500 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ca800e040 con 0x7f0cac0fbe80 2024-03-11T23:00:44.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.692+0000 7f0cb3ffc700 1 -- 172.21.15.153:0/616351500 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cac115540 con 0x7f0cac0fbe80 2024-03-11T23:00:44.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.692+0000 7f0ca27fc700 1 -- 172.21.15.153:0/616351500 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0ca800bcc0 con 0x7f0cac0fbe80 2024-03-11T23:00:44.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.692+0000 7f0ca27fc700 1 -- 172.21.15.153:0/616351500 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ca801c630 con 0x7f0cac0fbe80 2024-03-11T23:00:44.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.692+0000 7f0cb3ffc700 1 -- 172.21.15.153:0/616351500 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cac1159a0 con 0x7f0cac0fbe80 2024-03-11T23:00:44.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.693+0000 7f0cb3ffc700 1 -- 172.21.15.153:0/616351500 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cac04bed0 con 0x7f0cac0fbe80 2024-03-11T23:00:44.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.693+0000 7f0ca27fc700 1 -- 172.21.15.153:0/616351500 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f0ca80139f0 con 0x7f0cac0fbe80 2024-03-11T23:00:44.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.694+0000 7f0ca27fc700 1 --2- 172.21.15.153:0/616351500 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f0c980692a0 0x7f0c9806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:44.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.694+0000 7f0ca27fc700 1 -- 172.21.15.153:0/616351500 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f0ca8088630 con 0x7f0cac0fbe80 2024-03-11T23:00:44.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.694+0000 7f0cb1597700 1 --2- 172.21.15.153:0/616351500 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f0c980692a0 0x7f0c9806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:44.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.694+0000 7f0cb1597700 1 --2- 172.21.15.153:0/616351500 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f0c980692a0 0x7f0c9806b750 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c006fd0 tx=0x7f0c9c008040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:00:44.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.698+0000 7f0ca27fc700 1 -- 172.21.15.153:0/616351500 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f0ca8054910 con 0x7f0cac0fbe80 2024-03-11T23:00:44.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.883+0000 7f0cb3ffc700 1 -- 172.21.15.153:0/616351500 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f0cac104990 con 0x7f0cac0fbe80 2024-03-11T23:00:44.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.884+0000 7f0ca27fc700 1 -- 172.21.15.153:0/616351500 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7f0ca8057f30 con 0x7f0cac0fbe80 2024-03-11T23:00:44.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.885+0000 7f0cb3ffc700 1 -- 172.21.15.153:0/616351500 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f0c980692a0 msgr2=0x7f0c9806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:44.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.885+0000 7f0cb3ffc700 1 --2- 172.21.15.153:0/616351500 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f0c980692a0 0x7f0c9806b750 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c006fd0 tx=0x7f0c9c008040 comp rx=0 tx=0).stop 2024-03-11T23:00:44.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.886+0000 7f0cb3ffc700 1 -- 172.21.15.153:0/616351500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cac0fbe80 msgr2=0x7f0cac112ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:44.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.886+0000 7f0cb3ffc700 1 --2- 172.21.15.153:0/616351500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cac0fbe80 0x7f0cac112ab0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0ca800b3b0 tx=0x7f0ca8009e30 comp rx=0 tx=0).stop 2024-03-11T23:00:44.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.886+0000 7f0cb3ffc700 1 -- 172.21.15.153:0/616351500 shutdown_connections 2024-03-11T23:00:44.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.886+0000 7f0cb3ffc700 1 --2- 172.21.15.153:0/616351500 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0cac0fbe80 0x7f0cac112ab0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:44.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.886+0000 7f0cb3ffc700 1 --2- 172.21.15.153:0/616351500 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f0c980692a0 0x7f0c9806b750 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:44.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.886+0000 7f0cb3ffc700 1 --2- 172.21.15.153:0/616351500 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0cac0fd080 0x7f0cac115000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:44.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.886+0000 7f0cb3ffc700 1 -- 172.21.15.153:0/616351500 >> 172.21.15.153:0/616351500 conn(0x7f0cac0f7590 msgr2=0x7f0cac1002b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:00:44.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.886+0000 7f0cb3ffc700 1 -- 172.21.15.153:0/616351500 shutdown_connections 2024-03-11T23:00:44.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:44.886+0000 7f0cb3ffc700 1 -- 172.21.15.153:0/616351500 wait complete. 2024-03-11T23:00:45.494 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:45 smithi153 ceph-mon[28962]: pgmap v198: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 938 B/s wr, 5 op/s 2024-03-11T23:00:45.522 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'ceph config set global log_to_journald false --force' 2024-03-11T23:00:45.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:45 smithi163 ceph-mon[35584]: pgmap v198: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 938 B/s wr, 5 op/s 2024-03-11T23:00:45.789 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T23:00:47.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.395+0000 7f9a43763700 1 -- 172.21.15.153:0/3397301002 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a3c0fcfb0 msgr2=0x7f9a3c0fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:47.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.395+0000 7f9a43763700 1 --2- 172.21.15.153:0/3397301002 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a3c0fcfb0 0x7f9a3c0fd420 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c004940 tx=0x7f9a2c009e30 comp rx=0 tx=0).stop 2024-03-11T23:00:47.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.395+0000 7f9a43763700 1 -- 172.21.15.153:0/3397301002 shutdown_connections 2024-03-11T23:00:47.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.395+0000 7f9a43763700 1 --2- 172.21.15.153:0/3397301002 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a3c0fcfb0 0x7f9a3c0fd420 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:47.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.395+0000 7f9a43763700 1 --2- 172.21.15.153:0/3397301002 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9a3c0fbe40 0x7f9a3c0fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:47.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.395+0000 7f9a43763700 1 -- 172.21.15.153:0/3397301002 >> 172.21.15.153:0/3397301002 conn(0x7f9a3c0f7520 msgr2=0x7f9a3c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:00:47.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.395+0000 7f9a43763700 1 -- 172.21.15.153:0/3397301002 shutdown_connections 2024-03-11T23:00:47.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.395+0000 7f9a43763700 1 -- 172.21.15.153:0/3397301002 wait complete. 2024-03-11T23:00:47.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.396+0000 7f9a43763700 1 Processor -- start 2024-03-11T23:00:47.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.396+0000 7f9a43763700 1 -- start start 2024-03-11T23:00:47.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.396+0000 7f9a43763700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9a3c0fbe40 0x7f9a3c10d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:47.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.396+0000 7f9a43763700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a3c0fcfb0 0x7f9a3c10de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:47.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.396+0000 7f9a43763700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3c10e470 con 0x7f9a3c0fcfb0 2024-03-11T23:00:47.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.396+0000 7f9a43763700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a3c10ed10 con 0x7f9a3c0fbe40 2024-03-11T23:00:47.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.396+0000 7f9a40cfe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a3c0fcfb0 0x7f9a3c10de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:47.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.396+0000 7f9a40cfe700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a3c0fcfb0 0x7f9a3c10de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:32848/0 (socket says 172.21.15.153:32848) 2024-03-11T23:00:47.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.396+0000 7f9a40cfe700 1 -- 172.21.15.153:0/2501553560 learned_addr learned my addr 172.21.15.153:0/2501553560 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T23:00:47.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.396+0000 7f9a414ff700 1 --2- 172.21.15.153:0/2501553560 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9a3c0fbe40 0x7f9a3c10d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:47.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.396+0000 7f9a40cfe700 1 -- 172.21.15.153:0/2501553560 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9a3c0fbe40 msgr2=0x7f9a3c10d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:47.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.396+0000 7f9a40cfe700 1 --2- 172.21.15.153:0/2501553560 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9a3c0fbe40 0x7f9a3c10d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:47.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.396+0000 7f9a40cfe700 1 -- 172.21.15.153:0/2501553560 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a2c005040 con 0x7f9a3c0fcfb0 2024-03-11T23:00:47.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.396+0000 7f9a40cfe700 1 --2- 172.21.15.153:0/2501553560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a3c0fcfb0 0x7f9a3c10de70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c0093b0 tx=0x7f9a2c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:00:47.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.397+0000 7f9a31ffb700 1 -- 172.21.15.153:0/2501553560 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a2c00e040 con 0x7f9a3c0fcfb0 2024-03-11T23:00:47.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.397+0000 7f9a43763700 1 -- 172.21.15.153:0/2501553560 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a3c10ef90 con 0x7f9a3c0fcfb0 2024-03-11T23:00:47.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.397+0000 7f9a31ffb700 1 -- 172.21.15.153:0/2501553560 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9a2c00bcc0 con 0x7f9a3c0fcfb0 2024-03-11T23:00:47.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.397+0000 7f9a31ffb700 1 -- 172.21.15.153:0/2501553560 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9a2c01c630 con 0x7f9a3c0fcfb0 2024-03-11T23:00:47.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.397+0000 7f9a43763700 1 -- 172.21.15.153:0/2501553560 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a3c10f4d0 con 0x7f9a3c0fcfb0 2024-03-11T23:00:47.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.398+0000 7f9a43763700 1 -- 172.21.15.153:0/2501553560 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a3c04bed0 con 0x7f9a3c0fcfb0 2024-03-11T23:00:47.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.398+0000 7f9a31ffb700 1 -- 172.21.15.153:0/2501553560 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f9a2c0139f0 con 0x7f9a3c0fcfb0 2024-03-11T23:00:47.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.399+0000 7f9a31ffb700 1 --2- 172.21.15.153:0/2501553560 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f9a280692a0 0x7f9a2806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:47.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.399+0000 7f9a31ffb700 1 -- 172.21.15.153:0/2501553560 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f9a2c0885e0 con 0x7f9a3c0fcfb0 2024-03-11T23:00:47.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.399+0000 7f9a414ff700 1 --2- 172.21.15.153:0/2501553560 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f9a280692a0 0x7f9a2806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:47.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.399+0000 7f9a414ff700 1 --2- 172.21.15.153:0/2501553560 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f9a280692a0 0x7f9a2806b750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9a38007fd0 tx=0x7f9a38009040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:00:47.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.403+0000 7f9a31ffb700 1 -- 172.21.15.153:0/2501553560 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f9a2c058350 con 0x7f9a3c0fcfb0 2024-03-11T23:00:47.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:47 smithi153 ceph-mon[28962]: pgmap v199: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 938 B/s wr, 4 op/s 2024-03-11T23:00:47.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.593+0000 7f9a43763700 1 -- 172.21.15.153:0/2501553560 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f9a3c10f9f0 con 0x7f9a3c0fcfb0 2024-03-11T23:00:47.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.593+0000 7f9a31ffb700 1 -- 172.21.15.153:0/2501553560 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7f9a2c057ee0 con 0x7f9a3c0fcfb0 2024-03-11T23:00:47.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.595+0000 7f9a43763700 1 -- 172.21.15.153:0/2501553560 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f9a280692a0 msgr2=0x7f9a2806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:47.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.595+0000 7f9a43763700 1 --2- 172.21.15.153:0/2501553560 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f9a280692a0 0x7f9a2806b750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9a38007fd0 tx=0x7f9a38009040 comp rx=0 tx=0).stop 2024-03-11T23:00:47.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.595+0000 7f9a43763700 1 -- 172.21.15.153:0/2501553560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a3c0fcfb0 msgr2=0x7f9a3c10de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:47.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.595+0000 7f9a43763700 1 --2- 172.21.15.153:0/2501553560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a3c0fcfb0 0x7f9a3c10de70 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c0093b0 tx=0x7f9a2c009e30 comp rx=0 tx=0).stop 2024-03-11T23:00:47.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.595+0000 7f9a43763700 1 -- 172.21.15.153:0/2501553560 shutdown_connections 2024-03-11T23:00:47.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.595+0000 7f9a43763700 1 --2- 172.21.15.153:0/2501553560 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f9a280692a0 0x7f9a2806b750 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:47.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.595+0000 7f9a43763700 1 --2- 172.21.15.153:0/2501553560 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9a3c0fbe40 0x7f9a3c10d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:47.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.595+0000 7f9a43763700 1 --2- 172.21.15.153:0/2501553560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9a3c0fcfb0 0x7f9a3c10de70 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:47.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.595+0000 7f9a43763700 1 -- 172.21.15.153:0/2501553560 >> 172.21.15.153:0/2501553560 conn(0x7f9a3c0f7520 msgr2=0x7f9a3c065f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:00:47.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.595+0000 7f9a43763700 1 -- 172.21.15.153:0/2501553560 shutdown_connections 2024-03-11T23:00:47.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:47.596+0000 7f9a43763700 1 -- 172.21.15.153:0/2501553560 wait complete. 2024-03-11T23:00:47.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:47 smithi163 ceph-mon[35584]: pgmap v199: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 938 B/s wr, 4 op/s 2024-03-11T23:00:48.315 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-03-11T23:00:48.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:48 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:00:48.583 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T23:00:48.595 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:48 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:00:49.488 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:49 smithi163 ceph-mon[35584]: pgmap v200: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 938 B/s wr, 4 op/s 2024-03-11T23:00:49.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:49 smithi153 ceph-mon[28962]: pgmap v200: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 938 B/s wr, 4 op/s 2024-03-11T23:00:50.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:50.352+0000 7f43f83d3700 1 -- 172.21.15.153:0/4048808762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43f00fbe40 msgr2=0x7f43f00fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:50.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:50.352+0000 7f43f83d3700 1 --2- 172.21.15.153:0/4048808762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43f00fbe40 0x7f43f00fc250 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f43e8009ce0 tx=0x7f43e8004d10 comp rx=0 tx=0).stop 2024-03-11T23:00:50.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:50.352+0000 7f43f83d3700 1 -- 172.21.15.153:0/4048808762 shutdown_connections 2024-03-11T23:00:50.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:50.352+0000 7f43f83d3700 1 --2- 172.21.15.153:0/4048808762 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43f00fcfb0 0x7f43f00fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:50.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:50.352+0000 7f43f83d3700 1 --2- 172.21.15.153:0/4048808762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43f00fbe40 0x7f43f00fc250 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:50.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:50.352+0000 7f43f83d3700 1 -- 172.21.15.153:0/4048808762 >> 172.21.15.153:0/4048808762 conn(0x7f43f00f74e0 msgr2=0x7f43f00f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:00:50.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:50.352+0000 7f43f83d3700 1 -- 172.21.15.153:0/4048808762 shutdown_connections 2024-03-11T23:00:50.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:50.352+0000 7f43f83d3700 1 -- 172.21.15.153:0/4048808762 wait complete. 2024-03-11T23:00:50.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:50.352+0000 7f43f83d3700 1 Processor -- start 2024-03-11T23:00:50.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:50.353+0000 7f43f83d3700 1 -- start start 2024-03-11T23:00:50.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:50.353+0000 7f43f83d3700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43f00fbe40 0x7f43f010daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:50.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:50.353+0000 7f43f83d3700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43f00fcfb0 0x7f43f010dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:50.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:50.353+0000 7f43f83d3700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f010e5e0 con 0x7f43f00fbe40 2024-03-11T23:00:50.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:50.353+0000 7f43f83d3700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f010ee80 con 0x7f43f00fcfb0 2024-03-11T23:00:50.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:50.353+0000 7f43f616f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43f00fbe40 0x7f43f010daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:50.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:50.353+0000 7f43f616f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43f00fbe40 0x7f43f010daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:32866/0 (socket says 172.21.15.153:32866) 2024-03-11T23:00:50.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:50.353+0000 7f43f616f700 1 -- 172.21.15.153:0/2206321376 learned_addr learned my addr 172.21.15.153:0/2206321376 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T23:00:50.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:50.353+0000 7f43f596e700 1 --2- 172.21.15.153:0/2206321376 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43f00fcfb0 0x7f43f010dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:50.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:50.353+0000 7f43f616f700 1 -- 172.21.15.153:0/2206321376 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43f00fcfb0 msgr2=0x7f43f010dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:00:50.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:50.353+0000 7f43f616f700 1 --2- 172.21.15.153:0/2206321376 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43f00fcfb0 0x7f43f010dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:00:50.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:50.353+0000 7f43f616f700 1 -- 172.21.15.153:0/2206321376 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e8009940 con 0x7f43f00fbe40 2024-03-11T23:00:50.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:50.353+0000 7f43f616f700 1 --2- 172.21.15.153:0/2206321376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43f00fbe40 0x7f43f010daa0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f43e8015040 tx=0x7f43e8004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:00:50.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:50.353+0000 7f43e6ffd700 1 -- 172.21.15.153:0/2206321376 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43e800e040 con 0x7f43f00fbe40 2024-03-11T23:00:50.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:50.354+0000 7f43e6ffd700 1 -- 172.21.15.153:0/2206321376 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f43e80128e0 con 0x7f43f00fbe40 2024-03-11T23:00:50.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:50.354+0000 7f43f83d3700 1 -- 172.21.15.153:0/2206321376 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43f010f100 con 0x7f43f00fbe40 2024-03-11T23:00:50.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:50.354+0000 7f43e6ffd700 1 -- 172.21.15.153:0/2206321376 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43e801c940 con 0x7f43f00fbe40 2024-03-11T23:00:50.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:50.354+0000 7f43f83d3700 1 -- 172.21.15.153:0/2206321376 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43f010f640 con 0x7f43f00fbe40 2024-03-11T23:00:50.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:50.355+0000 7f43f83d3700 1 -- 172.21.15.153:0/2206321376 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43f004bed0 con 0x7f43f00fbe40 2024-03-11T23:00:50.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:50.355+0000 7f43e6ffd700 1 -- 172.21.15.153:0/2206321376 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f43e8012410 con 0x7f43f00fbe40 2024-03-11T23:00:50.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:50.355+0000 7f43e6ffd700 1 --2- 172.21.15.153:0/2206321376 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f43dc0692a0 0x7f43dc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:00:50.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:50.356+0000 7f43e6ffd700 1 -- 172.21.15.153:0/2206321376 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f43e8088ec0 con 0x7f43f00fbe40 2024-03-11T23:00:50.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:50.356+0000 7f43f596e700 1 --2- 172.21.15.153:0/2206321376 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f43dc0692a0 0x7f43dc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:00:50.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:50.356+0000 7f43f596e700 1 --2- 172.21.15.153:0/2206321376 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f43dc0692a0 0x7f43dc06b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f43e0007fd0 tx=0x7f43e0009040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:00:50.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:50.360+0000 7f43e6ffd700 1 -- 172.21.15.153:0/2206321376 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f43e8058c30 con 0x7f43f00fbe40 2024-03-11T23:00:50.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:00:50.560+0000 7f43f83d3700 1 -- 172.21.15.153:0/2206321376 --> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf", "daemon_types": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f43f0065dd0 con 0x7f43dc0692a0 2024-03-11T23:00:51.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:51 smithi163 ceph-mon[35584]: pgmap v201: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 5 op/s 2024-03-11T23:00:51.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:51 smithi163 ceph-mon[35584]: from='client.14530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:00:51.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:51 smithi153 ceph-mon[28962]: pgmap v201: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 5 op/s 2024-03-11T23:00:51.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:51 smithi153 ceph-mon[28962]: from='client.14530 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:00:53.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:53 smithi163 ceph-mon[35584]: pgmap v202: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 767 B/s wr, 4 op/s 2024-03-11T23:00:53.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:53 smithi153 ceph-mon[28962]: pgmap v202: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 767 B/s wr, 4 op/s 2024-03-11T23:00:55.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:55 smithi163 ceph-mon[35584]: pgmap v203: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-11T23:00:55.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:55 smithi153 ceph-mon[28962]: pgmap v203: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-11T23:00:57.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:57 smithi163 ceph-mon[35584]: pgmap v204: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 85 B/s wr, 3 op/s 2024-03-11T23:00:57.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:57 smithi153 ceph-mon[28962]: pgmap v204: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 85 B/s wr, 3 op/s 2024-03-11T23:00:59.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:00:59 smithi163 ceph-mon[35584]: pgmap v205: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 3 op/s 2024-03-11T23:00:59.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:00:59 smithi153 ceph-mon[28962]: pgmap v205: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 3 op/s 2024-03-11T23:01:01.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:01 smithi163 ceph-mon[35584]: pgmap v206: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 4 op/s 2024-03-11T23:01:01.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:01 smithi153 ceph-mon[28962]: pgmap v206: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 4 op/s 2024-03-11T23:01:03.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:03 smithi163 ceph-mon[35584]: pgmap v207: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-11T23:01:03.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:03 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:01:03.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:03 smithi153 ceph-mon[28962]: pgmap v207: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-11T23:01:03.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:03 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:01:05.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:05 smithi163 ceph-mon[35584]: pgmap v208: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-11T23:01:05.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:05 smithi153 ceph-mon[28962]: pgmap v208: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-11T23:01:07.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:07 smithi163 ceph-mon[35584]: pgmap v209: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-11T23:01:07.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:07 smithi153 ceph-mon[28962]: pgmap v209: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-11T23:01:09.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:09 smithi163 ceph-mon[35584]: pgmap v210: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-11T23:01:09.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:09 smithi153 ceph-mon[28962]: pgmap v210: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-11T23:01:11.737 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:11 smithi163 ceph-mon[35584]: pgmap v211: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-11T23:01:11.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:11 smithi153 ceph-mon[28962]: pgmap v211: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-11T23:01:13.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:13 smithi163 ceph-mon[35584]: pgmap v212: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-11T23:01:13.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:13 smithi153 ceph-mon[28962]: pgmap v212: 65 pgs: 65 active+clean; 586 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-11T23:01:15.737 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:15 smithi163 ceph-mon[35584]: pgmap v213: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 341 B/s wr, 4 op/s 2024-03-11T23:01:15.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:15 smithi153 ceph-mon[28962]: pgmap v213: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 341 B/s wr, 4 op/s 2024-03-11T23:01:17.737 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:17 smithi163 ceph-mon[35584]: pgmap v214: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-11T23:01:17.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:17 smithi153 ceph-mon[28962]: pgmap v214: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-11T23:01:18.487 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:18 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:01:18.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:18 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:01:19.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:19 smithi163 ceph-mon[35584]: pgmap v215: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 341 B/s wr, 4 op/s 2024-03-11T23:01:19.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:19 smithi153 ceph-mon[28962]: pgmap v215: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 341 B/s wr, 4 op/s 2024-03-11T23:01:21.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:21 smithi163 ceph-mon[35584]: pgmap v216: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 5 op/s 2024-03-11T23:01:21.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:21 smithi153 ceph-mon[28962]: pgmap v216: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 5 op/s 2024-03-11T23:01:23.737 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:23 smithi163 ceph-mon[35584]: pgmap v217: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 4 op/s 2024-03-11T23:01:23.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:23 smithi153 ceph-mon[28962]: pgmap v217: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 597 B/s wr, 4 op/s 2024-03-11T23:01:24.579 INFO:teuthology.orchestra.run.smithi153.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:01:24.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:24.575+0000 7f43e6ffd700 1 -- 172.21.15.153:0/2206321376 <== mgr.14204 v2:172.21.15.153:6800/29938766 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f43f0065dd0 con 0x7f43dc0692a0 2024-03-11T23:01:24.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:24.577+0000 7f43f83d3700 1 -- 172.21.15.153:0/2206321376 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f43dc0692a0 msgr2=0x7f43dc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:01:24.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:24.577+0000 7f43f83d3700 1 --2- 172.21.15.153:0/2206321376 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f43dc0692a0 0x7f43dc06b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f43e0007fd0 tx=0x7f43e0009040 comp rx=0 tx=0).stop 2024-03-11T23:01:24.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:24.577+0000 7f43f83d3700 1 -- 172.21.15.153:0/2206321376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43f00fbe40 msgr2=0x7f43f010daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:01:24.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:24.577+0000 7f43f83d3700 1 --2- 172.21.15.153:0/2206321376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43f00fbe40 0x7f43f010daa0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f43e8015040 tx=0x7f43e8004d10 comp rx=0 tx=0).stop 2024-03-11T23:01:24.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:24.577+0000 7f43f83d3700 1 -- 172.21.15.153:0/2206321376 shutdown_connections 2024-03-11T23:01:24.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:24.577+0000 7f43f83d3700 1 --2- 172.21.15.153:0/2206321376 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43f00fbe40 0x7f43f010daa0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:01:24.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:24.577+0000 7f43f83d3700 1 --2- 172.21.15.153:0/2206321376 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f43dc0692a0 0x7f43dc06b750 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:01:24.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:24.577+0000 7f43f83d3700 1 --2- 172.21.15.153:0/2206321376 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f43f00fcfb0 0x7f43f010dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:01:24.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:24.577+0000 7f43f83d3700 1 -- 172.21.15.153:0/2206321376 >> 172.21.15.153:0/2206321376 conn(0x7f43f00f74e0 msgr2=0x7f43f00656c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:01:24.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:24.577+0000 7f43f83d3700 1 -- 172.21.15.153:0/2206321376 shutdown_connections 2024-03-11T23:01:24.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:24.577+0000 7f43f83d3700 1 -- 172.21.15.153:0/2206321376 wait complete. 2024-03-11T23:01:25.185 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-11T23:01:25.510 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T23:01:25.829 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:25 smithi153 ceph-mon[28962]: pgmap v218: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-11T23:01:25.829 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:25 smithi153 ceph-mon[28962]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:01:25.829 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:25 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:01:25.829 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:25 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:01:25.829 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:25 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:01:25.829 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:25 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:01:25.830 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:25 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:01:25.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:25 smithi163 ceph-mon[35584]: pgmap v218: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.4 KiB/s rd, 682 B/s wr, 5 op/s 2024-03-11T23:01:25.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:25 smithi163 ceph-mon[35584]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:01:25.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:25 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:01:25.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:25 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:01:25.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:25 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T23:01:25.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:25 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T23:01:25.989 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:25 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:01:26.939 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:26 smithi153 ceph-mon[28962]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:01:26.987 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:26 smithi163 ceph-mon[35584]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T23:01:27.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.389+0000 7fef9f787700 1 -- 172.21.15.153:0/2624326914 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef980fda80 msgr2=0x7fef980ffe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:01:27.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.389+0000 7fef9f787700 1 --2- 172.21.15.153:0/2624326914 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef980fda80 0x7fef980ffe60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fef94004940 tx=0x7fef94009e30 comp rx=0 tx=0).stop 2024-03-11T23:01:27.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.389+0000 7fef9f787700 1 -- 172.21.15.153:0/2624326914 shutdown_connections 2024-03-11T23:01:27.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.389+0000 7fef9f787700 1 --2- 172.21.15.153:0/2624326914 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef981003a0 0x7fef98102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:01:27.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.389+0000 7fef9f787700 1 --2- 172.21.15.153:0/2624326914 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef980fda80 0x7fef980ffe60 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:01:27.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.389+0000 7fef9f787700 1 -- 172.21.15.153:0/2624326914 >> 172.21.15.153:0/2624326914 conn(0x7fef980f74e0 msgr2=0x7fef980f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:01:27.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.389+0000 7fef9f787700 1 -- 172.21.15.153:0/2624326914 shutdown_connections 2024-03-11T23:01:27.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.389+0000 7fef9f787700 1 -- 172.21.15.153:0/2624326914 wait complete. 2024-03-11T23:01:27.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.390+0000 7fef9f787700 1 Processor -- start 2024-03-11T23:01:27.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.390+0000 7fef9f787700 1 -- start start 2024-03-11T23:01:27.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.390+0000 7fef9f787700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef980fda80 0x7fef9810d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:01:27.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.390+0000 7fef9f787700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef981003a0 0x7fef9810de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:01:27.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.390+0000 7fef9f787700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9810e400 con 0x7fef980fda80 2024-03-11T23:01:27.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.390+0000 7fef9f787700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9810eca0 con 0x7fef981003a0 2024-03-11T23:01:27.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.390+0000 7fef9cd22700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef981003a0 0x7fef9810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:01:27.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.391+0000 7fef9cd22700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef981003a0 0x7fef9810de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.153:40944/0 (socket says 172.21.15.153:40944) 2024-03-11T23:01:27.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.391+0000 7fef9cd22700 1 -- 172.21.15.153:0/3907545675 learned_addr learned my addr 172.21.15.153:0/3907545675 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T23:01:27.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.391+0000 7fef9d523700 1 --2- 172.21.15.153:0/3907545675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef980fda80 0x7fef9810d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:01:27.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.391+0000 7fef9cd22700 1 -- 172.21.15.153:0/3907545675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef980fda80 msgr2=0x7fef9810d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:01:27.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.391+0000 7fef9cd22700 1 --2- 172.21.15.153:0/3907545675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef980fda80 0x7fef9810d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:01:27.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.391+0000 7fef9cd22700 1 -- 172.21.15.153:0/3907545675 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef94005040 con 0x7fef981003a0 2024-03-11T23:01:27.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.391+0000 7fef9d523700 1 --2- 172.21.15.153:0/3907545675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef980fda80 0x7fef9810d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T23:01:27.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.391+0000 7fef9cd22700 1 --2- 172.21.15.153:0/3907545675 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef981003a0 0x7fef9810de00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fef8800eb40 tx=0x7fef8800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:01:27.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.391+0000 7fef8dffb700 1 -- 172.21.15.153:0/3907545675 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef88007cc0 con 0x7fef981003a0 2024-03-11T23:01:27.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.391+0000 7fef9f787700 1 -- 172.21.15.153:0/3907545675 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef9810ef20 con 0x7fef981003a0 2024-03-11T23:01:27.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.391+0000 7fef8dffb700 1 -- 172.21.15.153:0/3907545675 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef8800f040 con 0x7fef981003a0 2024-03-11T23:01:27.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.392+0000 7fef9f787700 1 -- 172.21.15.153:0/3907545675 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef9810f460 con 0x7fef981003a0 2024-03-11T23:01:27.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.392+0000 7fef8dffb700 1 -- 172.21.15.153:0/3907545675 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef880092b0 con 0x7fef981003a0 2024-03-11T23:01:27.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.393+0000 7fef9f787700 1 -- 172.21.15.153:0/3907545675 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef98107a50 con 0x7fef981003a0 2024-03-11T23:01:27.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.394+0000 7fef8dffb700 1 -- 172.21.15.153:0/3907545675 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fef88009410 con 0x7fef981003a0 2024-03-11T23:01:27.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.395+0000 7fef8dffb700 1 --2- 172.21.15.153:0/3907545675 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fef840692a0 0x7fef8406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:01:27.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.395+0000 7fef8dffb700 1 -- 172.21.15.153:0/3907545675 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fef88086b40 con 0x7fef981003a0 2024-03-11T23:01:27.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.395+0000 7fef9d523700 1 --2- 172.21.15.153:0/3907545675 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fef840692a0 0x7fef8406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:01:27.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.396+0000 7fef9d523700 1 --2- 172.21.15.153:0/3907545675 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fef840692a0 0x7fef8406b750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fef94009b30 tx=0x7fef94009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:01:27.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.402+0000 7fef8dffb700 1 -- 172.21.15.153:0/3907545675 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7fef88056890 con 0x7fef981003a0 2024-03-11T23:01:27.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.660+0000 7fef9f787700 1 -- 172.21.15.153:0/3907545675 --> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fef9810f740 con 0x7fef840692a0 2024-03-11T23:01:27.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.661+0000 7fef8dffb700 1 -- 172.21.15.153:0/3907545675 <== mgr.14204 v2:172.21.15.153:6800/29938766 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7fef9810f740 con 0x7fef840692a0 2024-03-11T23:01:27.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.664+0000 7fef9f787700 1 -- 172.21.15.153:0/3907545675 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fef840692a0 msgr2=0x7fef8406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:01:27.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.664+0000 7fef9f787700 1 --2- 172.21.15.153:0/3907545675 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fef840692a0 0x7fef8406b750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fef94009b30 tx=0x7fef94009e30 comp rx=0 tx=0).stop 2024-03-11T23:01:27.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.664+0000 7fef9f787700 1 -- 172.21.15.153:0/3907545675 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef981003a0 msgr2=0x7fef9810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:01:27.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.664+0000 7fef9f787700 1 --2- 172.21.15.153:0/3907545675 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef981003a0 0x7fef9810de00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fef8800eb40 tx=0x7fef8800b040 comp rx=0 tx=0).stop 2024-03-11T23:01:27.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.664+0000 7fef9f787700 1 -- 172.21.15.153:0/3907545675 shutdown_connections 2024-03-11T23:01:27.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.664+0000 7fef9f787700 1 --2- 172.21.15.153:0/3907545675 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fef980fda80 0x7fef9810d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:01:27.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.664+0000 7fef9f787700 1 --2- 172.21.15.153:0/3907545675 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fef840692a0 0x7fef8406b750 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:01:27.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.664+0000 7fef9f787700 1 --2- 172.21.15.153:0/3907545675 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef981003a0 0x7fef9810de00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:01:27.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.664+0000 7fef9f787700 1 -- 172.21.15.153:0/3907545675 >> 172.21.15.153:0/3907545675 conn(0x7fef980f74e0 msgr2=0x7fef980f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:01:27.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.665+0000 7fef9f787700 1 -- 172.21.15.153:0/3907545675 shutdown_connections 2024-03-11T23:01:27.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.665+0000 7fef9f787700 1 -- 172.21.15.153:0/3907545675 wait complete. 2024-03-11T23:01:27.683 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-03-11T23:01:27.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.823+0000 7f6f5f3e4700 1 -- 172.21.15.153:0/4115756359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f580fcfb0 msgr2=0x7f6f580fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:01:27.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.823+0000 7f6f5f3e4700 1 --2- 172.21.15.153:0/4115756359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f580fcfb0 0x7f6f580fd420 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6f48004940 tx=0x7f6f48009e30 comp rx=0 tx=0).stop 2024-03-11T23:01:27.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.823+0000 7f6f5f3e4700 1 -- 172.21.15.153:0/4115756359 shutdown_connections 2024-03-11T23:01:27.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.823+0000 7f6f5f3e4700 1 --2- 172.21.15.153:0/4115756359 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f580fcfb0 0x7f6f580fd420 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:01:27.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.823+0000 7f6f5f3e4700 1 --2- 172.21.15.153:0/4115756359 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f580fbe40 0x7f6f580fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:01:27.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.823+0000 7f6f5f3e4700 1 -- 172.21.15.153:0/4115756359 >> 172.21.15.153:0/4115756359 conn(0x7f6f580f74e0 msgr2=0x7f6f580f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:01:27.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.823+0000 7f6f5f3e4700 1 -- 172.21.15.153:0/4115756359 shutdown_connections 2024-03-11T23:01:27.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.823+0000 7f6f5f3e4700 1 -- 172.21.15.153:0/4115756359 wait complete. 2024-03-11T23:01:27.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.824+0000 7f6f5f3e4700 1 Processor -- start 2024-03-11T23:01:27.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.824+0000 7f6f5f3e4700 1 -- start start 2024-03-11T23:01:27.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.824+0000 7f6f5f3e4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f580fbe40 0x7f6f5810c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:01:27.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.824+0000 7f6f5f3e4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f580fcfb0 0x7f6f5810cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:01:27.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.825+0000 7f6f5d180700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f580fbe40 0x7f6f5810c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:01:27.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.825+0000 7f6f5c97f700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f580fcfb0 0x7f6f5810cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:01:27.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.825+0000 7f6f5d180700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f580fbe40 0x7f6f5810c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.153:40966/0 (socket says 172.21.15.153:40966) 2024-03-11T23:01:27.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.825+0000 7f6f5d180700 1 -- 172.21.15.153:0/873581972 learned_addr learned my addr 172.21.15.153:0/873581972 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T23:01:27.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.825+0000 7f6f5f3e4700 1 -- 172.21.15.153:0/873581972 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5810d2e0 con 0x7f6f580fcfb0 2024-03-11T23:01:27.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.825+0000 7f6f5f3e4700 1 -- 172.21.15.153:0/873581972 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5810db80 con 0x7f6f580fbe40 2024-03-11T23:01:27.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.825+0000 7f6f5d180700 1 -- 172.21.15.153:0/873581972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f580fcfb0 msgr2=0x7f6f5810cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:01:27.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.825+0000 7f6f5d180700 1 --2- 172.21.15.153:0/873581972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f580fcfb0 0x7f6f5810cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:01:27.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.825+0000 7f6f5d180700 1 -- 172.21.15.153:0/873581972 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f48005040 con 0x7f6f580fbe40 2024-03-11T23:01:27.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.825+0000 7f6f5c97f700 1 --2- 172.21.15.153:0/873581972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f580fcfb0 0x7f6f5810cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-11T23:01:27.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.825+0000 7f6f5d180700 1 --2- 172.21.15.153:0/873581972 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f580fbe40 0x7f6f5810c7a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6f5400eb70 tx=0x7f6f5400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:01:27.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.826+0000 7f6f4dffb700 1 -- 172.21.15.153:0/873581972 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f54007cc0 con 0x7f6f580fbe40 2024-03-11T23:01:27.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.826+0000 7f6f4dffb700 1 -- 172.21.15.153:0/873581972 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6f5400f040 con 0x7f6f580fbe40 2024-03-11T23:01:27.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.826+0000 7f6f5f3e4700 1 -- 172.21.15.153:0/873581972 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f5810de00 con 0x7f6f580fbe40 2024-03-11T23:01:27.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.827+0000 7f6f4dffb700 1 -- 172.21.15.153:0/873581972 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f5400ca90 con 0x7f6f580fbe40 2024-03-11T23:01:27.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.827+0000 7f6f5f3e4700 1 -- 172.21.15.153:0/873581972 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f5810e280 con 0x7f6f580fbe40 2024-03-11T23:01:27.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.828+0000 7f6f5f3e4700 1 -- 172.21.15.153:0/873581972 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f5804bed0 con 0x7f6f580fbe40 2024-03-11T23:01:27.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.828+0000 7f6f4dffb700 1 -- 172.21.15.153:0/873581972 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f6f5400cbf0 con 0x7f6f580fbe40 2024-03-11T23:01:27.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.829+0000 7f6f4dffb700 1 --2- 172.21.15.153:0/873581972 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f6f440692a0 0x7f6f4406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:01:27.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.829+0000 7f6f4dffb700 1 -- 172.21.15.153:0/873581972 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f6f54086d70 con 0x7f6f580fbe40 2024-03-11T23:01:27.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.829+0000 7f6f5c97f700 1 --2- 172.21.15.153:0/873581972 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f6f440692a0 0x7f6f4406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:01:27.833 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:27 smithi153 ceph-mon[28962]: pgmap v219: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-11T23:01:27.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.829+0000 7f6f5c97f700 1 --2- 172.21.15.153:0/873581972 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f6f440692a0 0x7f6f4406b750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6f48009b30 tx=0x7f6f48009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:01:27.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:27.837+0000 7f6f4dffb700 1 -- 172.21.15.153:0/873581972 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f6f54056a40 con 0x7f6f580fbe40 2024-03-11T23:01:27.890 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:27 smithi163 ceph-mon[35584]: pgmap v219: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 341 B/s wr, 3 op/s 2024-03-11T23:01:28.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.245+0000 7f6f5f3e4700 1 -- 172.21.15.153:0/873581972 --> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f58101990 con 0x7f6f440692a0 2024-03-11T23:01:28.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.247+0000 7f6f4dffb700 1 -- 172.21.15.153:0/873581972 <== mgr.14204 v2:172.21.15.153:6800/29938766 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7f6f58101990 con 0x7f6f440692a0 2024-03-11T23:01:28.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.250+0000 7f6f5f3e4700 1 -- 172.21.15.153:0/873581972 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f6f440692a0 msgr2=0x7f6f4406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:01:28.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.250+0000 7f6f5f3e4700 1 --2- 172.21.15.153:0/873581972 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f6f440692a0 0x7f6f4406b750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6f48009b30 tx=0x7f6f48009e30 comp rx=0 tx=0).stop 2024-03-11T23:01:28.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.250+0000 7f6f5f3e4700 1 -- 172.21.15.153:0/873581972 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f580fbe40 msgr2=0x7f6f5810c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:01:28.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.250+0000 7f6f5f3e4700 1 --2- 172.21.15.153:0/873581972 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f580fbe40 0x7f6f5810c7a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6f5400eb70 tx=0x7f6f5400b040 comp rx=0 tx=0).stop 2024-03-11T23:01:28.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.250+0000 7f6f5f3e4700 1 -- 172.21.15.153:0/873581972 shutdown_connections 2024-03-11T23:01:28.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.250+0000 7f6f5f3e4700 1 --2- 172.21.15.153:0/873581972 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f6f440692a0 0x7f6f4406b750 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:01:28.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.250+0000 7f6f5f3e4700 1 --2- 172.21.15.153:0/873581972 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f580fbe40 0x7f6f5810c7a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:01:28.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.250+0000 7f6f5f3e4700 1 --2- 172.21.15.153:0/873581972 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6f580fcfb0 0x7f6f5810cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:01:28.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.250+0000 7f6f5f3e4700 1 -- 172.21.15.153:0/873581972 >> 172.21.15.153:0/873581972 conn(0x7f6f580f74e0 msgr2=0x7f6f58100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:01:28.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.251+0000 7f6f5f3e4700 1 -- 172.21.15.153:0/873581972 shutdown_connections 2024-03-11T23:01:28.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.251+0000 7f6f5f3e4700 1 -- 172.21.15.153:0/873581972 wait complete. 2024-03-11T23:01:28.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.415+0000 7f248dcad700 1 -- 172.21.15.153:0/2625464356 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24880fcfb0 msgr2=0x7f24880fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:01:28.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.415+0000 7f248dcad700 1 --2- 172.21.15.153:0/2625464356 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24880fcfb0 0x7f24880fd420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f247c004940 tx=0x7f247c009e30 comp rx=0 tx=0).stop 2024-03-11T23:01:28.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.415+0000 7f248dcad700 1 -- 172.21.15.153:0/2625464356 shutdown_connections 2024-03-11T23:01:28.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.415+0000 7f248dcad700 1 --2- 172.21.15.153:0/2625464356 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24880fcfb0 0x7f24880fd420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:01:28.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.415+0000 7f248dcad700 1 --2- 172.21.15.153:0/2625464356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24880fbe40 0x7f24880fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:01:28.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.415+0000 7f248dcad700 1 -- 172.21.15.153:0/2625464356 >> 172.21.15.153:0/2625464356 conn(0x7f24880f7520 msgr2=0x7f24880f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:01:28.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.415+0000 7f248dcad700 1 -- 172.21.15.153:0/2625464356 shutdown_connections 2024-03-11T23:01:28.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.415+0000 7f248dcad700 1 -- 172.21.15.153:0/2625464356 wait complete. 2024-03-11T23:01:28.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.416+0000 7f248dcad700 1 Processor -- start 2024-03-11T23:01:28.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.417+0000 7f248dcad700 1 -- start start 2024-03-11T23:01:28.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.417+0000 7f248dcad700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24880fbe40 0x7f248810c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:01:28.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.417+0000 7f248dcad700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24880fcfb0 0x7f248810cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:01:28.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.417+0000 7f2486ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24880fcfb0 0x7f248810cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:01:28.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.417+0000 7f2486ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24880fcfb0 0x7f248810cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54250/0 (socket says 172.21.15.153:54250) 2024-03-11T23:01:28.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.418+0000 7f24877fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24880fbe40 0x7f248810c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:01:28.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.418+0000 7f24877fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24880fbe40 0x7f248810c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.153:40980/0 (socket says 172.21.15.153:40980) 2024-03-11T23:01:28.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.418+0000 7f2486ffd700 1 -- 172.21.15.153:0/742952405 learned_addr learned my addr 172.21.15.153:0/742952405 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T23:01:28.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.418+0000 7f248dcad700 1 -- 172.21.15.153:0/742952405 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f248810d310 con 0x7f24880fcfb0 2024-03-11T23:01:28.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.418+0000 7f248dcad700 1 -- 172.21.15.153:0/742952405 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f248810dbb0 con 0x7f24880fbe40 2024-03-11T23:01:28.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.418+0000 7f2486ffd700 1 -- 172.21.15.153:0/742952405 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24880fbe40 msgr2=0x7f248810c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:01:28.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.418+0000 7f2486ffd700 1 --2- 172.21.15.153:0/742952405 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24880fbe40 0x7f248810c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:01:28.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.418+0000 7f2486ffd700 1 -- 172.21.15.153:0/742952405 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f247c005040 con 0x7f24880fcfb0 2024-03-11T23:01:28.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.419+0000 7f2486ffd700 1 --2- 172.21.15.153:0/742952405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24880fcfb0 0x7f248810cd10 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f247c000c00 tx=0x7f247c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:01:28.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.419+0000 7f246ffff700 1 -- 172.21.15.153:0/742952405 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f247c021070 con 0x7f24880fcfb0 2024-03-11T23:01:28.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.419+0000 7f248dcad700 1 -- 172.21.15.153:0/742952405 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f248810de30 con 0x7f24880fcfb0 2024-03-11T23:01:28.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.419+0000 7f246ffff700 1 -- 172.21.15.153:0/742952405 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f247c026410 con 0x7f24880fcfb0 2024-03-11T23:01:28.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.419+0000 7f248dcad700 1 -- 172.21.15.153:0/742952405 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f248810e250 con 0x7f24880fcfb0 2024-03-11T23:01:28.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.420+0000 7f248dcad700 1 -- 172.21.15.153:0/742952405 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f248804bed0 con 0x7f24880fcfb0 2024-03-11T23:01:28.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.421+0000 7f246ffff700 1 -- 172.21.15.153:0/742952405 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f247c00e070 con 0x7f24880fcfb0 2024-03-11T23:01:28.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.422+0000 7f246ffff700 1 -- 172.21.15.153:0/742952405 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f247c0269d0 con 0x7f24880fcfb0 2024-03-11T23:01:28.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.422+0000 7f246ffff700 1 --2- 172.21.15.153:0/742952405 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f24700692b0 0x7f247006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:01:28.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.422+0000 7f246ffff700 1 -- 172.21.15.153:0/742952405 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f247c088430 con 0x7f24880fcfb0 2024-03-11T23:01:28.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.422+0000 7f24877fe700 1 --2- 172.21.15.153:0/742952405 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f24700692b0 0x7f247006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:01:28.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.423+0000 7f24877fe700 1 --2- 172.21.15.153:0/742952405 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f24700692b0 0x7f247006b760 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2478004600 tx=0x7f247800b040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:01:28.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.431+0000 7f246ffff700 1 -- 172.21.15.153:0/742952405 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f247c0581a0 con 0x7f24880fcfb0 2024-03-11T23:01:28.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.826+0000 7f248dcad700 1 -- 172.21.15.153:0/742952405 --> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f248810e7f0 con 0x7f24700692b0 2024-03-11T23:01:28.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.844+0000 7f246ffff700 1 -- 172.21.15.153:0/742952405 <== mgr.14204 v2:172.21.15.153:6800/29938766 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7f248810e7f0 con 0x7f24700692b0 2024-03-11T23:01:28.849 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T23:01:28.849 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (5m) 60s ago 6m 25.5M - 0.25.0 c8568f914cd2 7b54ac6814f8 2024-03-11T23:01:28.849 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (6m) 60s ago 6m 6665k - 17.2.7-620-gc186d0ca a92afbb1c283 9a75910c50de 2024-03-11T23:01:28.850 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi163 smithi163 running (6m) 66s ago 6m 6660k - 17.2.7-620-gc186d0ca a92afbb1c283 8f8506abaa8f 2024-03-11T23:01:28.850 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (5m) 60s ago 6m 86.4M - 9.4.7 954c08fa6188 3fd24b741053 2024-03-11T23:01:28.850 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.qxjyxz smithi153 running (86s) 60s ago 85s 16.2M - 17.2.7-620-gc186d0ca a92afbb1c283 b2f49a3d6898 2024-03-11T23:01:28.850 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.yuzdaj smithi153 running (79s) 60s ago 79s 15.8M - 17.2.7-620-gc186d0ca a92afbb1c283 5d7909363070 2024-03-11T23:01:28.850 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi163.kzggfd smithi163 running (77s) 66s ago 76s 13.5M - 17.2.7-620-gc186d0ca a92afbb1c283 f9b67a225c35 2024-03-11T23:01:28.850 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi163.pmppga smithi163 running (83s) 66s ago 82s 11.3M - 17.2.7-620-gc186d0ca a92afbb1c283 2b0d7517a2a7 2024-03-11T23:01:28.850 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.nyocep smithi153 *:9283 running (8m) 60s ago 8m 481M - 17.2.7-620-gc186d0ca a92afbb1c283 fb9a27cf063c 2024-03-11T23:01:28.851 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi163.ivtqrh smithi163 *:8443,9283 running (5m) 66s ago 5m 437M - 17.2.7-620-gc186d0ca a92afbb1c283 8584d0f7d4fa 2024-03-11T23:01:28.851 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (8m) 60s ago 8m 47.0M 2048M 17.2.7-620-gc186d0ca a92afbb1c283 0dad3ee1fdff 2024-03-11T23:01:28.851 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi163 smithi163 running (5m) 66s ago 5m 36.6M 2048M 17.2.7-620-gc186d0ca a92afbb1c283 fadb9d7bcbe8 2024-03-11T23:01:28.851 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (6m) 60s ago 6m 16.0M - 1.5.0 0da6a335fe13 cddc4be66b19 2024-03-11T23:01:28.851 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi163 smithi163 *:9100 running (5m) 66s ago 5m 16.9M - 1.5.0 0da6a335fe13 0aa9766c249d 2024-03-11T23:01:28.851 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (5m) 60s ago 5m 47.3M 4288M 17.2.7-620-gc186d0ca a92afbb1c283 e1d999ae7d94 2024-03-11T23:01:28.851 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (4m) 60s ago 4m 45.0M 4288M 17.2.7-620-gc186d0ca a92afbb1c283 23e1e01405ed 2024-03-11T23:01:28.851 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (4m) 60s ago 4m 43.3M 4288M 17.2.7-620-gc186d0ca a92afbb1c283 74c3d447a069 2024-03-11T23:01:28.851 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi163 running (3m) 66s ago 3m 44.2M 5312M 17.2.7-620-gc186d0ca a92afbb1c283 2b06c7febdd0 2024-03-11T23:01:28.851 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi163 running (3m) 66s ago 3m 42.3M 5312M 17.2.7-620-gc186d0ca a92afbb1c283 6e1f1b2171fb 2024-03-11T23:01:28.851 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi163 running (2m) 66s ago 2m 41.6M 5312M 17.2.7-620-gc186d0ca a92afbb1c283 c5d707e483c5 2024-03-11T23:01:28.851 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi153 smithi153 *:9095 running (5m) 60s ago 6m 52.5M - 2.43.0 a07b618ecd1d 7a3323c1a837 2024-03-11T23:01:28.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.847+0000 7f248dcad700 1 -- 172.21.15.153:0/742952405 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f24700692b0 msgr2=0x7f247006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:01:28.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.847+0000 7f248dcad700 1 --2- 172.21.15.153:0/742952405 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f24700692b0 0x7f247006b760 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2478004600 tx=0x7f247800b040 comp rx=0 tx=0).stop 2024-03-11T23:01:28.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.847+0000 7f248dcad700 1 -- 172.21.15.153:0/742952405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24880fcfb0 msgr2=0x7f248810cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:01:28.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.847+0000 7f248dcad700 1 --2- 172.21.15.153:0/742952405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24880fcfb0 0x7f248810cd10 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f247c000c00 tx=0x7f247c009e30 comp rx=0 tx=0).stop 2024-03-11T23:01:28.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.848+0000 7f248dcad700 1 -- 172.21.15.153:0/742952405 shutdown_connections 2024-03-11T23:01:28.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.848+0000 7f248dcad700 1 --2- 172.21.15.153:0/742952405 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f24700692b0 0x7f247006b760 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:01:28.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.848+0000 7f248dcad700 1 --2- 172.21.15.153:0/742952405 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24880fbe40 0x7f248810c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:01:28.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.848+0000 7f248dcad700 1 --2- 172.21.15.153:0/742952405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f24880fcfb0 0x7f248810cd10 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:01:28.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.848+0000 7f248dcad700 1 -- 172.21.15.153:0/742952405 >> 172.21.15.153:0/742952405 conn(0x7f24880f7520 msgr2=0x7f2488100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:01:28.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.848+0000 7f248dcad700 1 -- 172.21.15.153:0/742952405 shutdown_connections 2024-03-11T23:01:28.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:28.848+0000 7f248dcad700 1 -- 172.21.15.153:0/742952405 wait complete. 2024-03-11T23:01:29.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.009+0000 7f2542988700 1 -- 172.21.15.153:0/3917821261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f253c0fcfb0 msgr2=0x7f253c0fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:01:29.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.009+0000 7f2542988700 1 --2- 172.21.15.153:0/3917821261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f253c0fcfb0 0x7f253c0fd420 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2530004940 tx=0x7f2530009e30 comp rx=0 tx=0).stop 2024-03-11T23:01:29.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.009+0000 7f2542988700 1 -- 172.21.15.153:0/3917821261 shutdown_connections 2024-03-11T23:01:29.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.009+0000 7f2542988700 1 --2- 172.21.15.153:0/3917821261 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f253c0fcfb0 0x7f253c0fd420 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:01:29.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.009+0000 7f2542988700 1 --2- 172.21.15.153:0/3917821261 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f253c0fbe40 0x7f253c0fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:01:29.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.009+0000 7f2542988700 1 -- 172.21.15.153:0/3917821261 >> 172.21.15.153:0/3917821261 conn(0x7f253c0f7500 msgr2=0x7f253c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:01:29.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.009+0000 7f2542988700 1 -- 172.21.15.153:0/3917821261 shutdown_connections 2024-03-11T23:01:29.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.009+0000 7f2542988700 1 -- 172.21.15.153:0/3917821261 wait complete. 2024-03-11T23:01:29.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.010+0000 7f2542988700 1 Processor -- start 2024-03-11T23:01:29.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.010+0000 7f2542988700 1 -- start start 2024-03-11T23:01:29.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.011+0000 7f2542988700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f253c0fbe40 0x7f253c10e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:01:29.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.011+0000 7f2542988700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f253c066c00 0x7f253c110aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:01:29.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.011+0000 7f2542988700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253c111050 con 0x7f253c066c00 2024-03-11T23:01:29.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.011+0000 7f253bfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f253c0fbe40 0x7f253c10e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:01:29.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.011+0000 7f253bfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f253c0fbe40 0x7f253c10e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.153:40998/0 (socket says 172.21.15.153:40998) 2024-03-11T23:01:29.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.011+0000 7f253bfff700 1 -- 172.21.15.153:0/1999777814 learned_addr learned my addr 172.21.15.153:0/1999777814 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T23:01:29.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.011+0000 7f253b7fe700 1 --2- 172.21.15.153:0/1999777814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f253c066c00 0x7f253c110aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:01:29.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.011+0000 7f2542988700 1 -- 172.21.15.153:0/1999777814 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f253c1111c0 con 0x7f253c0fbe40 2024-03-11T23:01:29.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.012+0000 7f253b7fe700 1 -- 172.21.15.153:0/1999777814 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f253c0fbe40 msgr2=0x7f253c10e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:01:29.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.012+0000 7f253b7fe700 1 --2- 172.21.15.153:0/1999777814 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f253c0fbe40 0x7f253c10e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:01:29.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.012+0000 7f253b7fe700 1 -- 172.21.15.153:0/1999777814 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2530005040 con 0x7f253c066c00 2024-03-11T23:01:29.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.012+0000 7f253b7fe700 1 --2- 172.21.15.153:0/1999777814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f253c066c00 0x7f253c110aa0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2530008000 tx=0x7f2530009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:01:29.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.013+0000 7f2538ff9700 1 -- 172.21.15.153:0/1999777814 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2530021070 con 0x7f253c066c00 2024-03-11T23:01:29.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.013+0000 7f2538ff9700 1 -- 172.21.15.153:0/1999777814 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2530013de0 con 0x7f253c066c00 2024-03-11T23:01:29.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.014+0000 7f2538ff9700 1 -- 172.21.15.153:0/1999777814 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f253000e070 con 0x7f253c066c00 2024-03-11T23:01:29.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.014+0000 7f2542988700 1 -- 172.21.15.153:0/1999777814 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f253c111440 con 0x7f253c066c00 2024-03-11T23:01:29.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.015+0000 7f2542988700 1 -- 172.21.15.153:0/1999777814 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f253c111900 con 0x7f253c066c00 2024-03-11T23:01:29.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.016+0000 7f2538ff9700 1 -- 172.21.15.153:0/1999777814 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f25300138a0 con 0x7f253c066c00 2024-03-11T23:01:29.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.016+0000 7f2542988700 1 -- 172.21.15.153:0/1999777814 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f253c04bed0 con 0x7f253c066c00 2024-03-11T23:01:29.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.017+0000 7f2538ff9700 1 --2- 172.21.15.153:0/1999777814 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f25240690a0 0x7f252406b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:01:29.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.017+0000 7f2538ff9700 1 -- 172.21.15.153:0/1999777814 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f2530087b80 con 0x7f253c066c00 2024-03-11T23:01:29.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.018+0000 7f253bfff700 1 --2- 172.21.15.153:0/1999777814 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f25240690a0 0x7f252406b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:01:29.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.018+0000 7f253bfff700 1 --2- 172.21.15.153:0/1999777814 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f25240690a0 0x7f252406b550 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f252c008fd0 tx=0x7f252c00b040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:01:29.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.026+0000 7f2538ff9700 1 -- 172.21.15.153:0/1999777814 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f2530053f30 con 0x7f253c066c00 2024-03-11T23:01:29.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.568+0000 7f2542988700 1 -- 172.21.15.153:0/1999777814 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f253c111d00 con 0x7f253c066c00 2024-03-11T23:01:29.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.569+0000 7f2538ff9700 1 -- 172.21.15.153:0/1999777814 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f2530057550 con 0x7f253c066c00 2024-03-11T23:01:29.574 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-03-11T23:01:29.574 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-03-11T23:01:29.574 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-620-gc186d0ca (c186d0ca78be18052defec2e0c064f7869af6d3f) quincy (stable)": 2 2024-03-11T23:01:29.574 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-11T23:01:29.574 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-03-11T23:01:29.574 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-620-gc186d0ca (c186d0ca78be18052defec2e0c064f7869af6d3f) quincy (stable)": 2 2024-03-11T23:01:29.574 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-11T23:01:29.574 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-03-11T23:01:29.574 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-620-gc186d0ca (c186d0ca78be18052defec2e0c064f7869af6d3f) quincy (stable)": 6 2024-03-11T23:01:29.575 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-11T23:01:29.575 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-03-11T23:01:29.575 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-620-gc186d0ca (c186d0ca78be18052defec2e0c064f7869af6d3f) quincy (stable)": 4 2024-03-11T23:01:29.575 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-11T23:01:29.575 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-03-11T23:01:29.575 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-620-gc186d0ca (c186d0ca78be18052defec2e0c064f7869af6d3f) quincy (stable)": 14 2024-03-11T23:01:29.575 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-11T23:01:29.575 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-03-11T23:01:29.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.572+0000 7f2542988700 1 -- 172.21.15.153:0/1999777814 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f25240690a0 msgr2=0x7f252406b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:01:29.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.572+0000 7f2542988700 1 --2- 172.21.15.153:0/1999777814 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f25240690a0 0x7f252406b550 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f252c008fd0 tx=0x7f252c00b040 comp rx=0 tx=0).stop 2024-03-11T23:01:29.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.572+0000 7f2542988700 1 -- 172.21.15.153:0/1999777814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f253c066c00 msgr2=0x7f253c110aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:01:29.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.572+0000 7f2542988700 1 --2- 172.21.15.153:0/1999777814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f253c066c00 0x7f253c110aa0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2530008000 tx=0x7f2530009e30 comp rx=0 tx=0).stop 2024-03-11T23:01:29.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.572+0000 7f2542988700 1 -- 172.21.15.153:0/1999777814 shutdown_connections 2024-03-11T23:01:29.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.572+0000 7f2542988700 1 --2- 172.21.15.153:0/1999777814 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f25240690a0 0x7f252406b550 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:01:29.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.572+0000 7f2542988700 1 --2- 172.21.15.153:0/1999777814 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f253c0fbe40 0x7f253c10e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:01:29.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.572+0000 7f2542988700 1 --2- 172.21.15.153:0/1999777814 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f253c066c00 0x7f253c110aa0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:01:29.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.572+0000 7f2542988700 1 -- 172.21.15.153:0/1999777814 >> 172.21.15.153:0/1999777814 conn(0x7f253c0f7500 msgr2=0x7f253c100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:01:29.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.572+0000 7f2542988700 1 -- 172.21.15.153:0/1999777814 shutdown_connections 2024-03-11T23:01:29.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.573+0000 7f2542988700 1 -- 172.21.15.153:0/1999777814 wait complete. 2024-03-11T23:01:29.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.719+0000 7f30428cb700 1 -- 172.21.15.153:0/2066921039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f303c0fcef0 msgr2=0x7f303c0fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:01:29.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.719+0000 7f30428cb700 1 --2- 172.21.15.153:0/2066921039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f303c0fcef0 0x7f303c0fd320 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f302c004940 tx=0x7f302c009e30 comp rx=0 tx=0).stop 2024-03-11T23:01:29.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.720+0000 7f30428cb700 1 -- 172.21.15.153:0/2066921039 shutdown_connections 2024-03-11T23:01:29.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.720+0000 7f30428cb700 1 --2- 172.21.15.153:0/2066921039 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f303c0fe0e0 0x7f303c0fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:01:29.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.720+0000 7f30428cb700 1 --2- 172.21.15.153:0/2066921039 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f303c0fcef0 0x7f303c0fd320 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:01:29.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.720+0000 7f30428cb700 1 -- 172.21.15.153:0/2066921039 >> 172.21.15.153:0/2066921039 conn(0x7f303c0f8620 msgr2=0x7f303c0faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:01:29.725 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:29 smithi153 ceph-mon[28962]: from='client.24315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:01:29.725 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:29 smithi153 ceph-mon[28962]: pgmap v220: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 341 B/s wr, 4 op/s 2024-03-11T23:01:29.725 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:29 smithi153 ceph-mon[28962]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:01:29.725 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:29 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/1999777814' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:01:29.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.720+0000 7f30428cb700 1 -- 172.21.15.153:0/2066921039 shutdown_connections 2024-03-11T23:01:29.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.720+0000 7f30428cb700 1 -- 172.21.15.153:0/2066921039 wait complete. 2024-03-11T23:01:29.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.721+0000 7f30428cb700 1 Processor -- start 2024-03-11T23:01:29.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.721+0000 7f30428cb700 1 -- start start 2024-03-11T23:01:29.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.721+0000 7f30428cb700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f303c0fcef0 0x7f303c10d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:01:29.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.721+0000 7f30428cb700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f303c0fe0e0 0x7f303c10de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:01:29.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.721+0000 7f30428cb700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303c10e430 con 0x7f303c0fe0e0 2024-03-11T23:01:29.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.721+0000 7f30428cb700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303c10ecd0 con 0x7f303c0fcef0 2024-03-11T23:01:29.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.722+0000 7f303bfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f303c0fcef0 0x7f303c10d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:01:29.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.722+0000 7f303bfff700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f303c0fcef0 0x7f303c10d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.153:41020/0 (socket says 172.21.15.153:41020) 2024-03-11T23:01:29.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.722+0000 7f303bfff700 1 -- 172.21.15.153:0/769812497 learned_addr learned my addr 172.21.15.153:0/769812497 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T23:01:29.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.722+0000 7f303b7fe700 1 --2- 172.21.15.153:0/769812497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f303c0fe0e0 0x7f303c10de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:01:29.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.722+0000 7f303bfff700 1 -- 172.21.15.153:0/769812497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f303c0fe0e0 msgr2=0x7f303c10de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:01:29.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.722+0000 7f303bfff700 1 --2- 172.21.15.153:0/769812497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f303c0fe0e0 0x7f303c10de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:01:29.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.722+0000 7f303bfff700 1 -- 172.21.15.153:0/769812497 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f302c005040 con 0x7f303c0fcef0 2024-03-11T23:01:29.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.722+0000 7f303b7fe700 1 --2- 172.21.15.153:0/769812497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f303c0fe0e0 0x7f303c10de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-11T23:01:29.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.722+0000 7f303bfff700 1 --2- 172.21.15.153:0/769812497 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f303c0fcef0 0x7f303c10d8f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f302c009fd0 tx=0x7f302c009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:01:29.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.722+0000 7f3038ff9700 1 -- 172.21.15.153:0/769812497 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f302c00e070 con 0x7f303c0fcef0 2024-03-11T23:01:29.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.723+0000 7f3038ff9700 1 -- 172.21.15.153:0/769812497 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f302c01c400 con 0x7f303c0fcef0 2024-03-11T23:01:29.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.723+0000 7f3038ff9700 1 -- 172.21.15.153:0/769812497 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f302c00b630 con 0x7f303c0fcef0 2024-03-11T23:01:29.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.723+0000 7f30428cb700 1 -- 172.21.15.153:0/769812497 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f303c10ef50 con 0x7f303c0fcef0 2024-03-11T23:01:29.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.723+0000 7f30428cb700 1 -- 172.21.15.153:0/769812497 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f303c10f430 con 0x7f303c0fcef0 2024-03-11T23:01:29.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.724+0000 7f30428cb700 1 -- 172.21.15.153:0/769812497 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f303c04bed0 con 0x7f303c0fcef0 2024-03-11T23:01:29.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.724+0000 7f3038ff9700 1 -- 172.21.15.153:0/769812497 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f302c01ca50 con 0x7f303c0fcef0 2024-03-11T23:01:29.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.724+0000 7f3038ff9700 1 --2- 172.21.15.153:0/769812497 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f30280692b0 0x7f302806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:01:29.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.724+0000 7f3038ff9700 1 -- 172.21.15.153:0/769812497 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f302c0887b0 con 0x7f303c0fcef0 2024-03-11T23:01:29.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.725+0000 7f303b7fe700 1 --2- 172.21.15.153:0/769812497 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f30280692b0 0x7f302806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:01:29.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.725+0000 7f303b7fe700 1 --2- 172.21.15.153:0/769812497 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f30280692b0 0x7f302806b760 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3024009fd0 tx=0x7f302400b040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:01:29.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:29.729+0000 7f3038ff9700 1 -- 172.21.15.153:0/769812497 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f302c0585c0 con 0x7f303c0fcef0 2024-03-11T23:01:29.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:29 smithi163 ceph-mon[35584]: from='client.24315 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:01:29.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:29 smithi163 ceph-mon[35584]: pgmap v220: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 341 B/s wr, 4 op/s 2024-03-11T23:01:29.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:29 smithi163 ceph-mon[35584]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:01:29.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:29 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/1999777814' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:01:30.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:30.074+0000 7f30428cb700 1 -- 172.21.15.153:0/769812497 --> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f303c10faf0 con 0x7f30280692b0 2024-03-11T23:01:30.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:30.075+0000 7f3038ff9700 1 -- 172.21.15.153:0/769812497 <== mgr.14204 v2:172.21.15.153:6800/29938766 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7f303c10faf0 con 0x7f30280692b0 2024-03-11T23:01:30.079 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-03-11T23:01:30.079 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf", 2024-03-11T23:01:30.079 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-03-11T23:01:30.079 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-11T23:01:30.079 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [], 2024-03-11T23:01:30.079 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "", 2024-03-11T23:01:30.079 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf image", 2024-03-11T23:01:30.079 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-03-11T23:01:30.079 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-03-11T23:01:30.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:30.077+0000 7f30428cb700 1 -- 172.21.15.153:0/769812497 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f30280692b0 msgr2=0x7f302806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:01:30.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:30.077+0000 7f30428cb700 1 --2- 172.21.15.153:0/769812497 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f30280692b0 0x7f302806b760 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3024009fd0 tx=0x7f302400b040 comp rx=0 tx=0).stop 2024-03-11T23:01:30.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:30.077+0000 7f30428cb700 1 -- 172.21.15.153:0/769812497 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f303c0fcef0 msgr2=0x7f303c10d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:01:30.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:30.077+0000 7f30428cb700 1 --2- 172.21.15.153:0/769812497 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f303c0fcef0 0x7f303c10d8f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f302c009fd0 tx=0x7f302c009e30 comp rx=0 tx=0).stop 2024-03-11T23:01:30.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:30.077+0000 7f30428cb700 1 -- 172.21.15.153:0/769812497 shutdown_connections 2024-03-11T23:01:30.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:30.077+0000 7f30428cb700 1 --2- 172.21.15.153:0/769812497 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f30280692b0 0x7f302806b760 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:01:30.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:30.077+0000 7f30428cb700 1 --2- 172.21.15.153:0/769812497 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f303c0fcef0 0x7f303c10d8f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:01:30.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:30.077+0000 7f30428cb700 1 --2- 172.21.15.153:0/769812497 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f303c0fe0e0 0x7f303c10de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:01:30.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:30.077+0000 7f30428cb700 1 -- 172.21.15.153:0/769812497 >> 172.21.15.153:0/769812497 conn(0x7f303c0f8620 msgr2=0x7f303c1013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:01:30.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:30.077+0000 7f30428cb700 1 -- 172.21.15.153:0/769812497 shutdown_connections 2024-03-11T23:01:30.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:01:30.077+0000 7f30428cb700 1 -- 172.21.15.153:0/769812497 wait complete. 2024-03-11T23:01:30.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:30 smithi163 ceph-mon[35584]: from='client.14542 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:01:31.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:30 smithi153 ceph-mon[28962]: from='client.14542 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:01:31.987 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:31 smithi163 ceph-mon[35584]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:01:31.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:31 smithi163 ceph-mon[35584]: pgmap v221: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-11T23:01:32.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:31 smithi153 ceph-mon[28962]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:01:32.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:31 smithi153 ceph-mon[28962]: pgmap v221: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-11T23:01:33.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:33 smithi153 ceph-mon[28962]: pgmap v222: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 3 op/s 2024-03-11T23:01:33.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:33 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:01:33.987 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:33 smithi163 ceph-mon[35584]: pgmap v222: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 3 op/s 2024-03-11T23:01:33.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:33 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:01:35.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:35 smithi163 ceph-mon[35584]: pgmap v223: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 4 op/s 2024-03-11T23:01:36.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:35 smithi153 ceph-mon[28962]: pgmap v223: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 85 B/s wr, 4 op/s 2024-03-11T23:01:37.987 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:37 smithi163 ceph-mon[35584]: pgmap v224: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-11T23:01:38.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:37 smithi153 ceph-mon[28962]: pgmap v224: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-11T23:01:38.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:38 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:01:38.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:38 smithi163 ceph-mon[35584]: Upgrade: Target is version 19.0.0-1607-g94a53970 (unknown) 2024-03-11T23:01:38.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:38 smithi163 ceph-mon[35584]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b'] 2024-03-11T23:01:38.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:38 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:01:38.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:38 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:01:38.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:38 smithi163 ceph-mon[35584]: Upgrade: Need to upgrade myself (mgr.smithi153.nyocep) 2024-03-11T23:01:39.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:38 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:01:39.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:38 smithi153 ceph-mon[28962]: Upgrade: Target is version 19.0.0-1607-g94a53970 (unknown) 2024-03-11T23:01:39.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:38 smithi153 ceph-mon[28962]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b'] 2024-03-11T23:01:39.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:38 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T23:01:39.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:38 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' 2024-03-11T23:01:39.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:38 smithi153 ceph-mon[28962]: Upgrade: Need to upgrade myself (mgr.smithi153.nyocep) 2024-03-11T23:01:39.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:39 smithi153 ceph-mon[28962]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b on smithi163 2024-03-11T23:01:39.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:39 smithi153 ceph-mon[28962]: pgmap v225: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-11T23:01:39.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:39 smithi163 ceph-mon[35584]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b on smithi163 2024-03-11T23:01:39.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:39 smithi163 ceph-mon[35584]: pgmap v225: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-11T23:01:41.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:41 smithi163 ceph-mon[35584]: pgmap v226: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-11T23:01:42.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:41 smithi153 ceph-mon[28962]: pgmap v226: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-11T23:01:43.987 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:43 smithi163 ceph-mon[35584]: pgmap v227: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-11T23:01:44.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:43 smithi153 ceph-mon[28962]: pgmap v227: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-11T23:01:45.987 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:45 smithi163 ceph-mon[35584]: pgmap v228: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-11T23:01:46.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:45 smithi153 ceph-mon[28962]: pgmap v228: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-11T23:01:47.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:47 smithi163 ceph-mon[35584]: pgmap v229: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-11T23:01:48.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:47 smithi153 ceph-mon[28962]: pgmap v229: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 3 op/s 2024-03-11T23:01:48.346 INFO:tasks.workunit.client.1.smithi163.stderr:Note: switching to 'fbcdfe674e56d1ea7a0810744e5915bb07961be5'. 2024-03-11T23:01:48.346 INFO:tasks.workunit.client.1.smithi163.stderr: 2024-03-11T23:01:48.346 INFO:tasks.workunit.client.1.smithi163.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-11T23:01:48.347 INFO:tasks.workunit.client.1.smithi163.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-11T23:01:48.347 INFO:tasks.workunit.client.1.smithi163.stderr:state without impacting any branches by switching back to a branch. 2024-03-11T23:01:48.347 INFO:tasks.workunit.client.1.smithi163.stderr: 2024-03-11T23:01:48.347 INFO:tasks.workunit.client.1.smithi163.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-11T23:01:48.347 INFO:tasks.workunit.client.1.smithi163.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-11T23:01:48.347 INFO:tasks.workunit.client.1.smithi163.stderr: 2024-03-11T23:01:48.347 INFO:tasks.workunit.client.1.smithi163.stderr: git switch -c 2024-03-11T23:01:48.347 INFO:tasks.workunit.client.1.smithi163.stderr: 2024-03-11T23:01:48.347 INFO:tasks.workunit.client.1.smithi163.stderr:Or undo this operation with: 2024-03-11T23:01:48.347 INFO:tasks.workunit.client.1.smithi163.stderr: 2024-03-11T23:01:48.347 INFO:tasks.workunit.client.1.smithi163.stderr: git switch - 2024-03-11T23:01:48.347 INFO:tasks.workunit.client.1.smithi163.stderr: 2024-03-11T23:01:48.347 INFO:tasks.workunit.client.1.smithi163.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-11T23:01:48.347 INFO:tasks.workunit.client.1.smithi163.stderr: 2024-03-11T23:01:48.347 INFO:tasks.workunit.client.1.smithi163.stderr:HEAD is now at fbcdfe674e5 Merge pull request #55941 from galsalomon66/wip-64694-squid 2024-03-11T23:01:48.349 DEBUG:teuthology.orchestra.run.smithi163:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-03-11T23:01:48.483 INFO:tasks.workunit.client.1.smithi163.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-11T23:01:48.485 INFO:tasks.workunit.client.1.smithi163.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-03-11T23:01:48.485 INFO:tasks.workunit.client.1.smithi163.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-11T23:01:48.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:48 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:01:49.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:48 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:01:49.212 INFO:tasks.workunit.client.1.smithi163.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-11T23:01:49.284 INFO:tasks.workunit.client.1.smithi163.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-11T23:01:49.325 INFO:tasks.workunit.client.1.smithi163.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-03-11T23:01:49.326 INFO:tasks.workunit.client.1.smithi163.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-03-11T23:01:49.326 INFO:tasks.workunit.client.1.smithi163.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-11T23:01:49.366 INFO:tasks.workunit.client.1.smithi163.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-03-11T23:01:49.370 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-11T23:01:49.370 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-03-11T23:01:49.400 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-03-11T23:01:49.402 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-03-11T23:01:49.402 DEBUG:teuthology.orchestra.run.smithi163:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fbcdfe674e56d1ea7a0810744e5915bb07961be5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-03-11T23:01:49.462 INFO:tasks.workunit.client.1.smithi163.stderr:+ mkdir -p fsstress 2024-03-11T23:01:49.464 INFO:tasks.workunit.client.1.smithi163.stderr:+ pushd fsstress 2024-03-11T23:01:49.465 INFO:tasks.workunit.client.1.smithi163.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-03-11T23:01:49.465 INFO:tasks.workunit.client.1.smithi163.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-03-11T23:01:49.671 INFO:tasks.workunit.client.0.smithi153.stderr:Note: switching to 'fbcdfe674e56d1ea7a0810744e5915bb07961be5'. 2024-03-11T23:01:49.671 INFO:tasks.workunit.client.0.smithi153.stderr: 2024-03-11T23:01:49.671 INFO:tasks.workunit.client.0.smithi153.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-11T23:01:49.671 INFO:tasks.workunit.client.0.smithi153.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-11T23:01:49.671 INFO:tasks.workunit.client.0.smithi153.stderr:state without impacting any branches by switching back to a branch. 2024-03-11T23:01:49.671 INFO:tasks.workunit.client.0.smithi153.stderr: 2024-03-11T23:01:49.671 INFO:tasks.workunit.client.0.smithi153.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-11T23:01:49.671 INFO:tasks.workunit.client.0.smithi153.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-11T23:01:49.671 INFO:tasks.workunit.client.0.smithi153.stderr: 2024-03-11T23:01:49.672 INFO:tasks.workunit.client.0.smithi153.stderr: git switch -c 2024-03-11T23:01:49.672 INFO:tasks.workunit.client.0.smithi153.stderr: 2024-03-11T23:01:49.672 INFO:tasks.workunit.client.0.smithi153.stderr:Or undo this operation with: 2024-03-11T23:01:49.672 INFO:tasks.workunit.client.0.smithi153.stderr: 2024-03-11T23:01:49.672 INFO:tasks.workunit.client.0.smithi153.stderr: git switch - 2024-03-11T23:01:49.672 INFO:tasks.workunit.client.0.smithi153.stderr: 2024-03-11T23:01:49.672 INFO:tasks.workunit.client.0.smithi153.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-11T23:01:49.672 INFO:tasks.workunit.client.0.smithi153.stderr: 2024-03-11T23:01:49.672 INFO:tasks.workunit.client.0.smithi153.stderr:HEAD is now at fbcdfe674e5 Merge pull request #55941 from galsalomon66/wip-64694-squid 2024-03-11T23:01:49.674 DEBUG:teuthology.orchestra.run.smithi153:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-11T23:01:49.708 INFO:tasks.workunit.client.0.smithi153.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-11T23:01:49.710 INFO:tasks.workunit.client.0.smithi153.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-11T23:01:49.710 INFO:tasks.workunit.client.0.smithi153.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-11T23:01:49.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:49 smithi153 ceph-mon[28962]: pgmap v230: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-11T23:01:49.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:49 smithi163 ceph-mon[35584]: pgmap v230: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-11T23:01:50.558 INFO:tasks.workunit.client.0.smithi153.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-11T23:01:50.628 INFO:tasks.workunit.client.0.smithi153.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-11T23:01:50.670 INFO:tasks.workunit.client.0.smithi153.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-11T23:01:50.671 INFO:tasks.workunit.client.0.smithi153.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-11T23:01:50.672 INFO:tasks.workunit.client.0.smithi153.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-11T23:01:50.711 INFO:tasks.workunit.client.0.smithi153.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-11T23:01:50.714 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-11T23:01:50.714 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-11T23:01:50.770 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-03-11T23:01:50.771 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-03-11T23:01:50.771 DEBUG:teuthology.orchestra.run.smithi153:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fbcdfe674e56d1ea7a0810744e5915bb07961be5 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-03-11T23:01:50.830 INFO:tasks.workunit.client.0.smithi153.stderr:+ mkdir -p fsstress 2024-03-11T23:01:50.832 INFO:tasks.workunit.client.0.smithi153.stderr:+ pushd fsstress 2024-03-11T23:01:50.832 INFO:tasks.workunit.client.0.smithi153.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-03-11T23:01:50.832 INFO:tasks.workunit.client.0.smithi153.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-03-11T23:01:51.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:51 smithi163 ceph-mon[35584]: pgmap v231: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-11T23:01:52.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:51 smithi153 ceph-mon[28962]: pgmap v231: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-11T23:01:53.987 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:53 smithi163 ceph-mon[35584]: pgmap v232: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-11T23:01:54.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:53 smithi153 ceph-mon[28962]: pgmap v232: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.1 KiB/s rd, 3 op/s 2024-03-11T23:01:54.476 INFO:tasks.workunit.client.1.smithi163.stderr:+ tar xzf ltp-full.tgz 2024-03-11T23:01:55.987 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:55 smithi163 ceph-mon[35584]: pgmap v233: 65 pgs: 65 active+clean; 616 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.0 KiB/s rd, 2.5 KiB/s wr, 5 op/s 2024-03-11T23:01:56.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:55 smithi153 ceph-mon[28962]: pgmap v233: 65 pgs: 65 active+clean; 616 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.0 KiB/s rd, 2.5 KiB/s wr, 5 op/s 2024-03-11T23:01:57.988 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:57 smithi163 ceph-mon[35584]: pgmap v234: 65 pgs: 65 active+clean; 1.1 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.6 KiB/s rd, 52 KiB/s wr, 11 op/s 2024-03-11T23:01:58.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:57 smithi153 ceph-mon[28962]: pgmap v234: 65 pgs: 65 active+clean; 1.1 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 3.6 KiB/s rd, 52 KiB/s wr, 11 op/s 2024-03-11T23:01:58.987 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:01:58 smithi163 ceph-mon[35584]: pgmap v235: 65 pgs: 65 active+clean; 5.4 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.0 KiB/s rd, 425 KiB/s wr, 20 op/s 2024-03-11T23:01:59.030 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:01:58 smithi153 ceph-mon[28962]: pgmap v235: 65 pgs: 65 active+clean; 5.4 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 4.0 KiB/s rd, 425 KiB/s wr, 20 op/s 2024-03-11T23:02:00.074 INFO:tasks.workunit.client.0.smithi153.stderr:+ tar xzf ltp-full.tgz 2024-03-11T23:02:00.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.186+0000 7f20c7ad4700 1 -- 172.21.15.153:0/2172852147 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20c0066c70 msgr2=0x7f20c00ffaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:02:00.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.186+0000 7f20c7ad4700 1 --2- 172.21.15.153:0/2172852147 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20c0066c70 0x7f20c00ffaf0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f20b0004940 tx=0x7f20b0009e30 comp rx=0 tx=0).stop 2024-03-11T23:02:00.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.186+0000 7f20c7ad4700 1 -- 172.21.15.153:0/2172852147 shutdown_connections 2024-03-11T23:02:00.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.186+0000 7f20c7ad4700 1 --2- 172.21.15.153:0/2172852147 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20c01000c0 0x7f20c0102520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:02:00.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.186+0000 7f20c7ad4700 1 --2- 172.21.15.153:0/2172852147 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20c0066c70 0x7f20c00ffaf0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:02:00.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.186+0000 7f20c7ad4700 1 -- 172.21.15.153:0/2172852147 >> 172.21.15.153:0/2172852147 conn(0x7f20c00f7520 msgr2=0x7f20c00f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:02:00.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.186+0000 7f20c7ad4700 1 -- 172.21.15.153:0/2172852147 shutdown_connections 2024-03-11T23:02:00.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.186+0000 7f20c7ad4700 1 -- 172.21.15.153:0/2172852147 wait complete. 2024-03-11T23:02:00.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.187+0000 7f20c7ad4700 1 Processor -- start 2024-03-11T23:02:00.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.187+0000 7f20c7ad4700 1 -- start start 2024-03-11T23:02:00.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.187+0000 7f20c7ad4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20c0066c70 0x7f20c01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:02:00.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.187+0000 7f20c7ad4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20c01000c0 0x7f20c0106c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:02:00.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.187+0000 7f20c7ad4700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c0107270 con 0x7f20c0066c70 2024-03-11T23:02:00.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.187+0000 7f20c7ad4700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c0107b10 con 0x7f20c01000c0 2024-03-11T23:02:00.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.187+0000 7f20c506f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20c01000c0 0x7f20c0106c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:02:00.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.187+0000 7f20c5870700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20c0066c70 0x7f20c01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:02:00.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.187+0000 7f20c506f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20c01000c0 0x7f20c0106c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.153:41118/0 (socket says 172.21.15.153:41118) 2024-03-11T23:02:00.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.187+0000 7f20c506f700 1 -- 172.21.15.153:0/2897575505 learned_addr learned my addr 172.21.15.153:0/2897575505 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T23:02:00.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.188+0000 7f20c506f700 1 -- 172.21.15.153:0/2897575505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20c0066c70 msgr2=0x7f20c01066e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:02:00.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.188+0000 7f20c506f700 1 --2- 172.21.15.153:0/2897575505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20c0066c70 0x7f20c01066e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:02:00.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.188+0000 7f20c506f700 1 -- 172.21.15.153:0/2897575505 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b0005040 con 0x7f20c01000c0 2024-03-11T23:02:00.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.188+0000 7f20c506f700 1 --2- 172.21.15.153:0/2897575505 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20c01000c0 0x7f20c0106c20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f20bc00eb40 tx=0x7f20bc00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:02:00.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.188+0000 7f20b67fc700 1 -- 172.21.15.153:0/2897575505 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20bc007cc0 con 0x7f20c01000c0 2024-03-11T23:02:00.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.188+0000 7f20c7ad4700 1 -- 172.21.15.153:0/2897575505 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20c0107d90 con 0x7f20c01000c0 2024-03-11T23:02:00.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.188+0000 7f20b67fc700 1 -- 172.21.15.153:0/2897575505 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f20bc00f040 con 0x7f20c01000c0 2024-03-11T23:02:00.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.188+0000 7f20b67fc700 1 -- 172.21.15.153:0/2897575505 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f20bc00cad0 con 0x7f20c01000c0 2024-03-11T23:02:00.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.188+0000 7f20c7ad4700 1 -- 172.21.15.153:0/2897575505 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20c01082d0 con 0x7f20c01000c0 2024-03-11T23:02:00.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.189+0000 7f20c7ad4700 1 -- 172.21.15.153:0/2897575505 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20c00f9050 con 0x7f20c01000c0 2024-03-11T23:02:00.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.190+0000 7f20b67fc700 1 -- 172.21.15.153:0/2897575505 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f20bc00cc30 con 0x7f20c01000c0 2024-03-11T23:02:00.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.191+0000 7f20b67fc700 1 --2- 172.21.15.153:0/2897575505 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f20ac0692a0 0x7f20ac06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:02:00.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.191+0000 7f20b67fc700 1 -- 172.21.15.153:0/2897575505 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f20bc086d80 con 0x7f20c01000c0 2024-03-11T23:02:00.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.191+0000 7f20c5870700 1 --2- 172.21.15.153:0/2897575505 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f20ac0692a0 0x7f20ac06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:02:00.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.191+0000 7f20c5870700 1 --2- 172.21.15.153:0/2897575505 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f20ac0692a0 0x7f20ac06b750 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f20b0000c00 tx=0x7f20b0009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:02:00.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.194+0000 7f20b67fc700 1 -- 172.21.15.153:0/2897575505 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f20bc053060 con 0x7f20c01000c0 2024-03-11T23:02:00.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.437+0000 7f20c7ad4700 1 -- 172.21.15.153:0/2897575505 --> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f20c002b340 con 0x7f20ac0692a0 2024-03-11T23:02:00.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.438+0000 7f20b67fc700 1 -- 172.21.15.153:0/2897575505 <== mgr.14204 v2:172.21.15.153:6800/29938766 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f20c002b340 con 0x7f20ac0692a0 2024-03-11T23:02:00.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.440+0000 7f20c7ad4700 1 -- 172.21.15.153:0/2897575505 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f20ac0692a0 msgr2=0x7f20ac06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:02:00.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.440+0000 7f20c7ad4700 1 --2- 172.21.15.153:0/2897575505 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f20ac0692a0 0x7f20ac06b750 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f20b0000c00 tx=0x7f20b0009e30 comp rx=0 tx=0).stop 2024-03-11T23:02:00.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.440+0000 7f20c7ad4700 1 -- 172.21.15.153:0/2897575505 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20c01000c0 msgr2=0x7f20c0106c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:02:00.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.440+0000 7f20c7ad4700 1 --2- 172.21.15.153:0/2897575505 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20c01000c0 0x7f20c0106c20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f20bc00eb40 tx=0x7f20bc00b040 comp rx=0 tx=0).stop 2024-03-11T23:02:00.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.440+0000 7f20c7ad4700 1 -- 172.21.15.153:0/2897575505 shutdown_connections 2024-03-11T23:02:00.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.440+0000 7f20c7ad4700 1 --2- 172.21.15.153:0/2897575505 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20c0066c70 0x7f20c01066e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:02:00.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.440+0000 7f20c7ad4700 1 --2- 172.21.15.153:0/2897575505 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f20ac0692a0 0x7f20ac06b750 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:02:00.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.440+0000 7f20c7ad4700 1 --2- 172.21.15.153:0/2897575505 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20c01000c0 0x7f20c0106c20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:02:00.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.440+0000 7f20c7ad4700 1 -- 172.21.15.153:0/2897575505 >> 172.21.15.153:0/2897575505 conn(0x7f20c00f7520 msgr2=0x7f20c0100d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:02:00.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.441+0000 7f20c7ad4700 1 -- 172.21.15.153:0/2897575505 shutdown_connections 2024-03-11T23:02:00.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.441+0000 7f20c7ad4700 1 -- 172.21.15.153:0/2897575505 wait complete. 2024-03-11T23:02:00.459 INFO:teuthology.orchestra.run.smithi153.stdout:true 2024-03-11T23:02:00.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.591+0000 7fce297d4700 1 -- 172.21.15.153:0/3849921417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce240fcfb0 msgr2=0x7fce240fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:02:00.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.591+0000 7fce297d4700 1 --2- 172.21.15.153:0/3849921417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce240fcfb0 0x7fce240fd420 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fce0c004940 tx=0x7fce0c009e30 comp rx=0 tx=0).stop 2024-03-11T23:02:00.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.591+0000 7fce297d4700 1 -- 172.21.15.153:0/3849921417 shutdown_connections 2024-03-11T23:02:00.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.591+0000 7fce297d4700 1 --2- 172.21.15.153:0/3849921417 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce240fcfb0 0x7fce240fd420 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:02:00.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.591+0000 7fce297d4700 1 --2- 172.21.15.153:0/3849921417 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce240fbe40 0x7fce240fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:02:00.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.591+0000 7fce297d4700 1 -- 172.21.15.153:0/3849921417 >> 172.21.15.153:0/3849921417 conn(0x7fce240f74e0 msgr2=0x7fce240f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:02:00.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.591+0000 7fce297d4700 1 -- 172.21.15.153:0/3849921417 shutdown_connections 2024-03-11T23:02:00.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.591+0000 7fce297d4700 1 -- 172.21.15.153:0/3849921417 wait complete. 2024-03-11T23:02:00.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.592+0000 7fce297d4700 1 Processor -- start 2024-03-11T23:02:00.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.592+0000 7fce297d4700 1 -- start start 2024-03-11T23:02:00.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.592+0000 7fce297d4700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce240fbe40 0x7fce24110630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:02:00.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.592+0000 7fce297d4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce240fcfb0 0x7fce24112b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:02:00.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.592+0000 7fce297d4700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce24066460 con 0x7fce240fbe40 2024-03-11T23:02:00.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.592+0000 7fce297d4700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce240665a0 con 0x7fce240fcfb0 2024-03-11T23:02:00.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.592+0000 7fce22ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce240fbe40 0x7fce24110630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:02:00.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.592+0000 7fce22ffd700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce240fbe40 0x7fce24110630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50944/0 (socket says 172.21.15.153:50944) 2024-03-11T23:02:00.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.592+0000 7fce227fc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce240fcfb0 0x7fce24112b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:02:00.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.592+0000 7fce22ffd700 1 -- 172.21.15.153:0/920425712 learned_addr learned my addr 172.21.15.153:0/920425712 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T23:02:00.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.593+0000 7fce22ffd700 1 -- 172.21.15.153:0/920425712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce240fcfb0 msgr2=0x7fce24112b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:02:00.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.593+0000 7fce22ffd700 1 --2- 172.21.15.153:0/920425712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce240fcfb0 0x7fce24112b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:02:00.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.593+0000 7fce22ffd700 1 -- 172.21.15.153:0/920425712 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce0c005040 con 0x7fce240fbe40 2024-03-11T23:02:00.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.593+0000 7fce22ffd700 1 --2- 172.21.15.153:0/920425712 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce240fbe40 0x7fce24110630 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fce1c00ec30 tx=0x7fce1c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:02:00.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.593+0000 7fce1b7fe700 1 -- 172.21.15.153:0/920425712 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce1c007b30 con 0x7fce240fbe40 2024-03-11T23:02:00.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.593+0000 7fce297d4700 1 -- 172.21.15.153:0/920425712 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce24113180 con 0x7fce240fbe40 2024-03-11T23:02:00.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.593+0000 7fce1b7fe700 1 -- 172.21.15.153:0/920425712 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce1c00f040 con 0x7fce240fbe40 2024-03-11T23:02:00.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.593+0000 7fce1b7fe700 1 -- 172.21.15.153:0/920425712 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce1c00cc90 con 0x7fce240fbe40 2024-03-11T23:02:00.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.593+0000 7fce297d4700 1 -- 172.21.15.153:0/920425712 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce241136c0 con 0x7fce240fbe40 2024-03-11T23:02:00.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.594+0000 7fce297d4700 1 -- 172.21.15.153:0/920425712 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce2404bed0 con 0x7fce240fbe40 2024-03-11T23:02:00.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.595+0000 7fce1b7fe700 1 -- 172.21.15.153:0/920425712 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fce1c009070 con 0x7fce240fbe40 2024-03-11T23:02:00.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.595+0000 7fce1b7fe700 1 --2- 172.21.15.153:0/920425712 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fce100692b0 0x7fce1006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:02:00.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.595+0000 7fce1b7fe700 1 -- 172.21.15.153:0/920425712 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fce1c087660 con 0x7fce240fbe40 2024-03-11T23:02:00.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.595+0000 7fce227fc700 1 --2- 172.21.15.153:0/920425712 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fce100692b0 0x7fce1006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:02:00.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.596+0000 7fce227fc700 1 --2- 172.21.15.153:0/920425712 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fce100692b0 0x7fce1006b760 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fce0c013740 tx=0x7fce0c009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:02:00.607 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.604+0000 7fce1b7fe700 1 -- 172.21.15.153:0/920425712 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7fce1c053940 con 0x7fce240fbe40 2024-03-11T23:02:00.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.872+0000 7fce297d4700 1 -- 172.21.15.153:0/920425712 --> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fce24101990 con 0x7fce100692b0 2024-03-11T23:02:00.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.873+0000 7fce1b7fe700 1 -- 172.21.15.153:0/920425712 <== mgr.14204 v2:172.21.15.153:6800/29938766 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7fce24101990 con 0x7fce100692b0 2024-03-11T23:02:00.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.875+0000 7fce297d4700 1 -- 172.21.15.153:0/920425712 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fce100692b0 msgr2=0x7fce1006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:02:00.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.875+0000 7fce297d4700 1 --2- 172.21.15.153:0/920425712 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fce100692b0 0x7fce1006b760 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fce0c013740 tx=0x7fce0c009e30 comp rx=0 tx=0).stop 2024-03-11T23:02:00.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.875+0000 7fce297d4700 1 -- 172.21.15.153:0/920425712 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce240fbe40 msgr2=0x7fce24110630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:02:00.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.875+0000 7fce297d4700 1 --2- 172.21.15.153:0/920425712 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce240fbe40 0x7fce24110630 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fce1c00ec30 tx=0x7fce1c00b040 comp rx=0 tx=0).stop 2024-03-11T23:02:00.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.875+0000 7fce297d4700 1 -- 172.21.15.153:0/920425712 shutdown_connections 2024-03-11T23:02:00.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.875+0000 7fce297d4700 1 --2- 172.21.15.153:0/920425712 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce240fbe40 0x7fce24110630 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:02:00.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.875+0000 7fce297d4700 1 --2- 172.21.15.153:0/920425712 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fce100692b0 0x7fce1006b760 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:02:00.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.875+0000 7fce297d4700 1 --2- 172.21.15.153:0/920425712 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fce240fcfb0 0x7fce24112b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:02:00.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.875+0000 7fce297d4700 1 -- 172.21.15.153:0/920425712 >> 172.21.15.153:0/920425712 conn(0x7fce240f74e0 msgr2=0x7fce24100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:02:00.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.875+0000 7fce297d4700 1 -- 172.21.15.153:0/920425712 shutdown_connections 2024-03-11T23:02:00.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.875+0000 7fce297d4700 1 -- 172.21.15.153:0/920425712 wait complete. 2024-03-11T23:02:00.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.983+0000 7f45eb5fc700 1 -- 172.21.15.153:0/1081278231 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45e40fda80 msgr2=0x7f45e40ffe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:02:00.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.983+0000 7f45eb5fc700 1 --2- 172.21.15.153:0/1081278231 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45e40fda80 0x7f45e40ffe60 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f45e0004940 tx=0x7f45e0009e30 comp rx=0 tx=0).stop 2024-03-11T23:02:00.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.983+0000 7f45eb5fc700 1 -- 172.21.15.153:0/1081278231 shutdown_connections 2024-03-11T23:02:00.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.984+0000 7f45eb5fc700 1 --2- 172.21.15.153:0/1081278231 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45e41003a0 0x7f45e4102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:02:00.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.984+0000 7f45eb5fc700 1 --2- 172.21.15.153:0/1081278231 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45e40fda80 0x7f45e40ffe60 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:02:00.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.984+0000 7f45eb5fc700 1 -- 172.21.15.153:0/1081278231 >> 172.21.15.153:0/1081278231 conn(0x7f45e40f74e0 msgr2=0x7f45e40f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:02:00.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.984+0000 7f45eb5fc700 1 -- 172.21.15.153:0/1081278231 shutdown_connections 2024-03-11T23:02:00.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.984+0000 7f45eb5fc700 1 -- 172.21.15.153:0/1081278231 wait complete. 2024-03-11T23:02:00.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.984+0000 7f45eb5fc700 1 Processor -- start 2024-03-11T23:02:00.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.984+0000 7f45eb5fc700 1 -- start start 2024-03-11T23:02:00.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.984+0000 7f45eb5fc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45e40fda80 0x7f45e40fd510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:02:00.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.985+0000 7f45eb5fc700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45e41003a0 0x7f45e40fb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:02:00.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.985+0000 7f45eb5fc700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e40fbf90 con 0x7f45e41003a0 2024-03-11T23:02:00.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.985+0000 7f45eb5fc700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e40fc100 con 0x7f45e40fda80 2024-03-11T23:02:00.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.985+0000 7f45e8b97700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45e41003a0 0x7f45e40fb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:02:00.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.985+0000 7f45e8b97700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45e41003a0 0x7f45e40fb990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50960/0 (socket says 172.21.15.153:50960) 2024-03-11T23:02:00.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.985+0000 7f45e9398700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45e40fda80 0x7f45e40fd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:02:00.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.985+0000 7f45e8b97700 1 -- 172.21.15.153:0/3352602822 learned_addr learned my addr 172.21.15.153:0/3352602822 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T23:02:00.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.985+0000 7f45e8b97700 1 -- 172.21.15.153:0/3352602822 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45e40fda80 msgr2=0x7f45e40fd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:02:00.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.985+0000 7f45e8b97700 1 --2- 172.21.15.153:0/3352602822 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45e40fda80 0x7f45e40fd510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:02:00.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.985+0000 7f45e8b97700 1 -- 172.21.15.153:0/3352602822 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e0005040 con 0x7f45e41003a0 2024-03-11T23:02:00.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.985+0000 7f45e8b97700 1 --2- 172.21.15.153:0/3352602822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45e41003a0 0x7f45e40fb990 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f45d400eb40 tx=0x7f45d400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:02:00.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.985+0000 7f45d9ffb700 1 -- 172.21.15.153:0/3352602822 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45d4007cc0 con 0x7f45e41003a0 2024-03-11T23:02:00.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.985+0000 7f45d9ffb700 1 -- 172.21.15.153:0/3352602822 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f45d400f040 con 0x7f45e41003a0 2024-03-11T23:02:00.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.985+0000 7f45eb5fc700 1 -- 172.21.15.153:0/3352602822 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45e40fc380 con 0x7f45e41003a0 2024-03-11T23:02:00.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.985+0000 7f45d9ffb700 1 -- 172.21.15.153:0/3352602822 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f45d400ca10 con 0x7f45e41003a0 2024-03-11T23:02:00.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.986+0000 7f45eb5fc700 1 -- 172.21.15.153:0/3352602822 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45e41158f0 con 0x7f45e41003a0 2024-03-11T23:02:00.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.987+0000 7f45eb5fc700 1 -- 172.21.15.153:0/3352602822 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45e4107a50 con 0x7f45e41003a0 2024-03-11T23:02:00.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.987+0000 7f45d9ffb700 1 -- 172.21.15.153:0/3352602822 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f45d400cc80 con 0x7f45e41003a0 2024-03-11T23:02:00.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.988+0000 7f45d9ffb700 1 --2- 172.21.15.153:0/3352602822 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f45d00692a0 0x7f45d006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:02:00.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.988+0000 7f45d9ffb700 1 -- 172.21.15.153:0/3352602822 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f45d4086dd0 con 0x7f45e41003a0 2024-03-11T23:02:00.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.988+0000 7f45e9398700 1 --2- 172.21.15.153:0/3352602822 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f45d00692a0 0x7f45d006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:02:00.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.988+0000 7f45e9398700 1 --2- 172.21.15.153:0/3352602822 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f45d00692a0 0x7f45d006b750 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f45e0000c00 tx=0x7f45e0009e30 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:02:00.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:00.992+0000 7f45d9ffb700 1 -- 172.21.15.153:0/3352602822 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f45d404dd20 con 0x7f45e41003a0 2024-03-11T23:02:01.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.207+0000 7f45eb5fc700 1 -- 172.21.15.153:0/3352602822 --> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f45e402b310 con 0x7f45d00692a0 2024-03-11T23:02:01.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.226+0000 7f45d9ffb700 1 -- 172.21.15.153:0/3352602822 <== mgr.14204 v2:172.21.15.153:6800/29938766 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3410 (secure 0 0 0) 0x7f45e402b310 con 0x7f45d00692a0 2024-03-11T23:02:01.230 INFO:teuthology.orchestra.run.smithi153.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T23:02:01.230 INFO:teuthology.orchestra.run.smithi153.stdout:alertmanager.smithi153 smithi153 *:9093,9094 running (5m) 92s ago 7m 25.5M - 0.25.0 c8568f914cd2 7b54ac6814f8 2024-03-11T23:02:01.231 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi153 smithi153 running (7m) 92s ago 7m 6665k - 17.2.7-620-gc186d0ca a92afbb1c283 9a75910c50de 2024-03-11T23:02:01.231 INFO:teuthology.orchestra.run.smithi153.stdout:crash.smithi163 smithi163 running (6m) 98s ago 6m 6660k - 17.2.7-620-gc186d0ca a92afbb1c283 8f8506abaa8f 2024-03-11T23:02:01.231 INFO:teuthology.orchestra.run.smithi153.stdout:grafana.smithi153 smithi153 *:3000 running (5m) 92s ago 7m 86.4M - 9.4.7 954c08fa6188 3fd24b741053 2024-03-11T23:02:01.231 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.qxjyxz smithi153 running (118s) 92s ago 117s 16.2M - 17.2.7-620-gc186d0ca a92afbb1c283 b2f49a3d6898 2024-03-11T23:02:01.231 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi153.yuzdaj smithi153 running (112s) 92s ago 111s 15.8M - 17.2.7-620-gc186d0ca a92afbb1c283 5d7909363070 2024-03-11T23:02:01.231 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi163.kzggfd smithi163 running (109s) 98s ago 109s 13.5M - 17.2.7-620-gc186d0ca a92afbb1c283 f9b67a225c35 2024-03-11T23:02:01.231 INFO:teuthology.orchestra.run.smithi153.stdout:mds.cephfs.smithi163.pmppga smithi163 running (115s) 98s ago 115s 11.3M - 17.2.7-620-gc186d0ca a92afbb1c283 2b0d7517a2a7 2024-03-11T23:02:01.231 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi153.nyocep smithi153 *:9283 running (8m) 92s ago 8m 481M - 17.2.7-620-gc186d0ca a92afbb1c283 fb9a27cf063c 2024-03-11T23:02:01.231 INFO:teuthology.orchestra.run.smithi153.stdout:mgr.smithi163.ivtqrh smithi163 *:8443,9283 running (6m) 98s ago 6m 437M - 17.2.7-620-gc186d0ca a92afbb1c283 8584d0f7d4fa 2024-03-11T23:02:01.231 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi153 smithi153 running (8m) 92s ago 8m 47.0M 2048M 17.2.7-620-gc186d0ca a92afbb1c283 0dad3ee1fdff 2024-03-11T23:02:01.231 INFO:teuthology.orchestra.run.smithi153.stdout:mon.smithi163 smithi163 running (6m) 98s ago 6m 36.6M 2048M 17.2.7-620-gc186d0ca a92afbb1c283 fadb9d7bcbe8 2024-03-11T23:02:01.231 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi153 smithi153 *:9100 running (7m) 92s ago 7m 16.0M - 1.5.0 0da6a335fe13 cddc4be66b19 2024-03-11T23:02:01.231 INFO:teuthology.orchestra.run.smithi153.stdout:node-exporter.smithi163 smithi163 *:9100 running (6m) 98s ago 6m 16.9M - 1.5.0 0da6a335fe13 0aa9766c249d 2024-03-11T23:02:01.232 INFO:teuthology.orchestra.run.smithi153.stdout:osd.0 smithi153 running (5m) 92s ago 5m 47.3M 4288M 17.2.7-620-gc186d0ca a92afbb1c283 e1d999ae7d94 2024-03-11T23:02:01.232 INFO:teuthology.orchestra.run.smithi153.stdout:osd.1 smithi153 running (5m) 92s ago 5m 45.0M 4288M 17.2.7-620-gc186d0ca a92afbb1c283 23e1e01405ed 2024-03-11T23:02:01.232 INFO:teuthology.orchestra.run.smithi153.stdout:osd.2 smithi153 running (4m) 92s ago 4m 43.3M 4288M 17.2.7-620-gc186d0ca a92afbb1c283 74c3d447a069 2024-03-11T23:02:01.232 INFO:teuthology.orchestra.run.smithi153.stdout:osd.3 smithi163 running (4m) 98s ago 4m 44.2M 5312M 17.2.7-620-gc186d0ca a92afbb1c283 2b06c7febdd0 2024-03-11T23:02:01.232 INFO:teuthology.orchestra.run.smithi153.stdout:osd.4 smithi163 running (3m) 98s ago 3m 42.3M 5312M 17.2.7-620-gc186d0ca a92afbb1c283 6e1f1b2171fb 2024-03-11T23:02:01.232 INFO:teuthology.orchestra.run.smithi153.stdout:osd.5 smithi163 running (3m) 98s ago 3m 41.6M 5312M 17.2.7-620-gc186d0ca a92afbb1c283 c5d707e483c5 2024-03-11T23:02:01.232 INFO:teuthology.orchestra.run.smithi153.stdout:prometheus.smithi153 smithi153 *:9095 running (5m) 92s ago 6m 52.5M - 2.43.0 a07b618ecd1d 7a3323c1a837 2024-03-11T23:02:01.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.228+0000 7f45eb5fc700 1 -- 172.21.15.153:0/3352602822 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f45d00692a0 msgr2=0x7f45d006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:02:01.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.228+0000 7f45eb5fc700 1 --2- 172.21.15.153:0/3352602822 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f45d00692a0 0x7f45d006b750 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f45e0000c00 tx=0x7f45e0009e30 comp rx=0 tx=0).stop 2024-03-11T23:02:01.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.228+0000 7f45eb5fc700 1 -- 172.21.15.153:0/3352602822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45e41003a0 msgr2=0x7f45e40fb990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:02:01.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.228+0000 7f45eb5fc700 1 --2- 172.21.15.153:0/3352602822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45e41003a0 0x7f45e40fb990 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f45d400eb40 tx=0x7f45d400b040 comp rx=0 tx=0).stop 2024-03-11T23:02:01.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.228+0000 7f45eb5fc700 1 -- 172.21.15.153:0/3352602822 shutdown_connections 2024-03-11T23:02:01.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.228+0000 7f45eb5fc700 1 --2- 172.21.15.153:0/3352602822 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f45d00692a0 0x7f45d006b750 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:02:01.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.228+0000 7f45eb5fc700 1 --2- 172.21.15.153:0/3352602822 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45e40fda80 0x7f45e40fd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:02:01.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.228+0000 7f45eb5fc700 1 --2- 172.21.15.153:0/3352602822 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45e41003a0 0x7f45e40fb990 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:02:01.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.228+0000 7f45eb5fc700 1 -- 172.21.15.153:0/3352602822 >> 172.21.15.153:0/3352602822 conn(0x7f45e40f74e0 msgr2=0x7f45e40f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:02:01.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.228+0000 7f45eb5fc700 1 -- 172.21.15.153:0/3352602822 shutdown_connections 2024-03-11T23:02:01.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.228+0000 7f45eb5fc700 1 -- 172.21.15.153:0/3352602822 wait complete. 2024-03-11T23:02:01.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.332+0000 7f60079ba700 1 -- 172.21.15.153:0/1863190373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60000fbe40 msgr2=0x7f60000fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:02:01.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.332+0000 7f60079ba700 1 --2- 172.21.15.153:0/1863190373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60000fbe40 0x7f60000fc250 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc004940 tx=0x7f5ffc009e30 comp rx=0 tx=0).stop 2024-03-11T23:02:01.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.332+0000 7f60079ba700 1 -- 172.21.15.153:0/1863190373 shutdown_connections 2024-03-11T23:02:01.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.332+0000 7f60079ba700 1 --2- 172.21.15.153:0/1863190373 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60000fcfb0 0x7f60000fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:02:01.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.332+0000 7f60079ba700 1 --2- 172.21.15.153:0/1863190373 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60000fbe40 0x7f60000fc250 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:02:01.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.332+0000 7f60079ba700 1 -- 172.21.15.153:0/1863190373 >> 172.21.15.153:0/1863190373 conn(0x7f60000f74e0 msgr2=0x7f60000f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:02:01.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.333+0000 7f60079ba700 1 -- 172.21.15.153:0/1863190373 shutdown_connections 2024-03-11T23:02:01.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.333+0000 7f60079ba700 1 -- 172.21.15.153:0/1863190373 wait complete. 2024-03-11T23:02:01.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.333+0000 7f60079ba700 1 Processor -- start 2024-03-11T23:02:01.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.333+0000 7f60079ba700 1 -- start start 2024-03-11T23:02:01.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.333+0000 7f60079ba700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60000fbe40 0x7f600010fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:02:01.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.333+0000 7f60079ba700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60000fcfb0 0x7f600010ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:02:01.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.333+0000 7f60079ba700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6000110580 con 0x7f60000fbe40 2024-03-11T23:02:01.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.333+0000 7f60079ba700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6000110e20 con 0x7f60000fcfb0 2024-03-11T23:02:01.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.333+0000 7f6005756700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60000fbe40 0x7f600010fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:02:01.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.333+0000 7f6005756700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60000fbe40 0x7f600010fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50980/0 (socket says 172.21.15.153:50980) 2024-03-11T23:02:01.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.333+0000 7f6005756700 1 -- 172.21.15.153:0/1727973934 learned_addr learned my addr 172.21.15.153:0/1727973934 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T23:02:01.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.334+0000 7f6004f55700 1 --2- 172.21.15.153:0/1727973934 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60000fcfb0 0x7f600010ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:02:01.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.334+0000 7f6005756700 1 -- 172.21.15.153:0/1727973934 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60000fcfb0 msgr2=0x7f600010ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:02:01.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.334+0000 7f6005756700 1 --2- 172.21.15.153:0/1727973934 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60000fcfb0 0x7f600010ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:02:01.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.334+0000 7f6005756700 1 -- 172.21.15.153:0/1727973934 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ffc005040 con 0x7f60000fbe40 2024-03-11T23:02:01.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.334+0000 7f6005756700 1 --2- 172.21.15.153:0/1727973934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60000fbe40 0x7f600010fa40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc000c00 tx=0x7f5ffc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:02:01.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.334+0000 7f5ff5ffb700 1 -- 172.21.15.153:0/1727973934 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ffc021070 con 0x7f60000fbe40 2024-03-11T23:02:01.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.334+0000 7f60079ba700 1 -- 172.21.15.153:0/1727973934 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60001110a0 con 0x7f60000fbe40 2024-03-11T23:02:01.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.334+0000 7f5ff5ffb700 1 -- 172.21.15.153:0/1727973934 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5ffc01c400 con 0x7f60000fbe40 2024-03-11T23:02:01.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.334+0000 7f5ff5ffb700 1 -- 172.21.15.153:0/1727973934 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ffc00e070 con 0x7f60000fbe40 2024-03-11T23:02:01.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.334+0000 7f60079ba700 1 -- 172.21.15.153:0/1727973934 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6000111580 con 0x7f60000fbe40 2024-03-11T23:02:01.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.336+0000 7f60079ba700 1 -- 172.21.15.153:0/1727973934 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f600004bed0 con 0x7f60000fbe40 2024-03-11T23:02:01.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.336+0000 7f5ff5ffb700 1 -- 172.21.15.153:0/1727973934 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f5ffc01ca50 con 0x7f60000fbe40 2024-03-11T23:02:01.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.337+0000 7f5ff5ffb700 1 --2- 172.21.15.153:0/1727973934 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f5fec0692a0 0x7f5fec06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:02:01.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.337+0000 7f5ff5ffb700 1 -- 172.21.15.153:0/1727973934 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7f5ffc088300 con 0x7f60000fbe40 2024-03-11T23:02:01.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.337+0000 7f6004f55700 1 --2- 172.21.15.153:0/1727973934 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f5fec0692a0 0x7f5fec06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:02:01.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.337+0000 7f6004f55700 1 --2- 172.21.15.153:0/1727973934 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f5fec0692a0 0x7f5fec06b750 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5ff0009b80 tx=0x7f5ff000b040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:02:01.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.345+0000 7f5ff5ffb700 1 -- 172.21.15.153:0/1727973934 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7f5ffc058110 con 0x7f60000fbe40 2024-03-11T23:02:01.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:02:01 smithi153 ceph-mon[28962]: pgmap v236: 65 pgs: 65 active+clean; 29 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 684 KiB/s rd, 2.4 MiB/s wr, 46 op/s 2024-03-11T23:02:01.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:02:01 smithi153 ceph-mon[28962]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:02:01.737 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:02:01 smithi163 ceph-mon[35584]: pgmap v236: 65 pgs: 65 active+clean; 29 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 684 KiB/s rd, 2.4 MiB/s wr, 46 op/s 2024-03-11T23:02:01.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:02:01 smithi163 ceph-mon[35584]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:02:01.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.780+0000 7f60079ba700 1 -- 172.21.15.153:0/1727973934 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6000111bd0 con 0x7f60000fbe40 2024-03-11T23:02:01.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.781+0000 7f5ff5ffb700 1 -- 172.21.15.153:0/1727973934 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f5ffc057ca0 con 0x7f60000fbe40 2024-03-11T23:02:01.785 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-03-11T23:02:01.785 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": { 2024-03-11T23:02:01.785 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-620-gc186d0ca (c186d0ca78be18052defec2e0c064f7869af6d3f) quincy (stable)": 2 2024-03-11T23:02:01.785 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-11T23:02:01.785 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": { 2024-03-11T23:02:01.785 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-620-gc186d0ca (c186d0ca78be18052defec2e0c064f7869af6d3f) quincy (stable)": 2 2024-03-11T23:02:01.785 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-11T23:02:01.785 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": { 2024-03-11T23:02:01.785 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-620-gc186d0ca (c186d0ca78be18052defec2e0c064f7869af6d3f) quincy (stable)": 6 2024-03-11T23:02:01.785 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-11T23:02:01.786 INFO:teuthology.orchestra.run.smithi153.stdout: "mds": { 2024-03-11T23:02:01.786 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-620-gc186d0ca (c186d0ca78be18052defec2e0c064f7869af6d3f) quincy (stable)": 4 2024-03-11T23:02:01.786 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-03-11T23:02:01.786 INFO:teuthology.orchestra.run.smithi153.stdout: "overall": { 2024-03-11T23:02:01.786 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph version 17.2.7-620-gc186d0ca (c186d0ca78be18052defec2e0c064f7869af6d3f) quincy (stable)": 14 2024-03-11T23:02:01.786 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-03-11T23:02:01.786 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-03-11T23:02:01.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.783+0000 7f60079ba700 1 -- 172.21.15.153:0/1727973934 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f5fec0692a0 msgr2=0x7f5fec06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:02:01.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.783+0000 7f60079ba700 1 --2- 172.21.15.153:0/1727973934 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f5fec0692a0 0x7f5fec06b750 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5ff0009b80 tx=0x7f5ff000b040 comp rx=0 tx=0).stop 2024-03-11T23:02:01.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.783+0000 7f60079ba700 1 -- 172.21.15.153:0/1727973934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60000fbe40 msgr2=0x7f600010fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:02:01.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.783+0000 7f60079ba700 1 --2- 172.21.15.153:0/1727973934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60000fbe40 0x7f600010fa40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc000c00 tx=0x7f5ffc009e30 comp rx=0 tx=0).stop 2024-03-11T23:02:01.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.783+0000 7f60079ba700 1 -- 172.21.15.153:0/1727973934 shutdown_connections 2024-03-11T23:02:01.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.783+0000 7f60079ba700 1 --2- 172.21.15.153:0/1727973934 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60000fbe40 0x7f600010fa40 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:02:01.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.783+0000 7f60079ba700 1 --2- 172.21.15.153:0/1727973934 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7f5fec0692a0 0x7f5fec06b750 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:02:01.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.783+0000 7f60079ba700 1 --2- 172.21.15.153:0/1727973934 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60000fcfb0 0x7f600010ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:02:01.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.783+0000 7f60079ba700 1 -- 172.21.15.153:0/1727973934 >> 172.21.15.153:0/1727973934 conn(0x7f60000f74e0 msgr2=0x7f60000656c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:02:01.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.783+0000 7f60079ba700 1 -- 172.21.15.153:0/1727973934 shutdown_connections 2024-03-11T23:02:01.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.783+0000 7f60079ba700 1 -- 172.21.15.153:0/1727973934 wait complete. 2024-03-11T23:02:01.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.892+0000 7fa04f99b700 1 -- 172.21.15.153:0/3528863495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0480fda80 msgr2=0x7fa0480ffe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:02:01.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.892+0000 7fa04f99b700 1 --2- 172.21.15.153:0/3528863495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0480fda80 0x7fa0480ffe60 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fa044004940 tx=0x7fa044009e30 comp rx=0 tx=0).stop 2024-03-11T23:02:01.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.892+0000 7fa04f99b700 1 -- 172.21.15.153:0/3528863495 shutdown_connections 2024-03-11T23:02:01.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.892+0000 7fa04f99b700 1 --2- 172.21.15.153:0/3528863495 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa0481003a0 0x7fa048102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:02:01.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.892+0000 7fa04f99b700 1 --2- 172.21.15.153:0/3528863495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0480fda80 0x7fa0480ffe60 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:02:01.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.892+0000 7fa04f99b700 1 -- 172.21.15.153:0/3528863495 >> 172.21.15.153:0/3528863495 conn(0x7fa0480f74e0 msgr2=0x7fa0480f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:02:01.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.892+0000 7fa04f99b700 1 -- 172.21.15.153:0/3528863495 shutdown_connections 2024-03-11T23:02:01.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.892+0000 7fa04f99b700 1 -- 172.21.15.153:0/3528863495 wait complete. 2024-03-11T23:02:01.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.893+0000 7fa04f99b700 1 Processor -- start 2024-03-11T23:02:01.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.893+0000 7fa04f99b700 1 -- start start 2024-03-11T23:02:01.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.893+0000 7fa04f99b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa0480fda80 0x7fa04810d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:02:01.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.893+0000 7fa04f99b700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0481003a0 0x7fa04810de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:02:01.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.893+0000 7fa04f99b700 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04810e400 con 0x7fa0481003a0 2024-03-11T23:02:01.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.893+0000 7fa04f99b700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04810eca0 con 0x7fa0480fda80 2024-03-11T23:02:01.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.893+0000 7fa04d737700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa0480fda80 0x7fa04810d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:02:01.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.893+0000 7fa04cf36700 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0481003a0 0x7fa04810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:02:01.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.893+0000 7fa04d737700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa0480fda80 0x7fa04810d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.153:41388/0 (socket says 172.21.15.153:41388) 2024-03-11T23:02:01.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.893+0000 7fa04d737700 1 -- 172.21.15.153:0/165191848 learned_addr learned my addr 172.21.15.153:0/165191848 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-03-11T23:02:01.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.894+0000 7fa04d737700 1 -- 172.21.15.153:0/165191848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0481003a0 msgr2=0x7fa04810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:02:01.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.894+0000 7fa04d737700 1 --2- 172.21.15.153:0/165191848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0481003a0 0x7fa04810de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:02:01.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.894+0000 7fa04d737700 1 -- 172.21.15.153:0/165191848 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa044005040 con 0x7fa0480fda80 2024-03-11T23:02:01.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.894+0000 7fa04d737700 1 --2- 172.21.15.153:0/165191848 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa0480fda80 0x7fa04810d8c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa044000c00 tx=0x7fa044009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:02:01.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.894+0000 7fa03dffb700 1 -- 172.21.15.153:0/165191848 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa044021070 con 0x7fa0480fda80 2024-03-11T23:02:01.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.894+0000 7fa03dffb700 1 -- 172.21.15.153:0/165191848 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa04401c400 con 0x7fa0480fda80 2024-03-11T23:02:01.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.894+0000 7fa04f99b700 1 -- 172.21.15.153:0/165191848 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa04810ef20 con 0x7fa0480fda80 2024-03-11T23:02:01.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.894+0000 7fa03dffb700 1 -- 172.21.15.153:0/165191848 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa04400e070 con 0x7fa0480fda80 2024-03-11T23:02:01.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.894+0000 7fa04f99b700 1 -- 172.21.15.153:0/165191848 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa04810f400 con 0x7fa0480fda80 2024-03-11T23:02:01.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.896+0000 7fa04f99b700 1 -- 172.21.15.153:0/165191848 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa048107a50 con 0x7fa0480fda80 2024-03-11T23:02:01.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.896+0000 7fa03dffb700 1 -- 172.21.15.153:0/165191848 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fa04401ca50 con 0x7fa0480fda80 2024-03-11T23:02:01.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.896+0000 7fa03dffb700 1 --2- 172.21.15.153:0/165191848 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fa0340692a0 0x7fa03406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-11T23:02:01.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.897+0000 7fa03dffb700 1 -- 172.21.15.153:0/165191848 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5550+0+0 (secure 0 0 0) 0x7fa0440883e0 con 0x7fa0480fda80 2024-03-11T23:02:01.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.897+0000 7fa04cf36700 1 --2- 172.21.15.153:0/165191848 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fa0340692a0 0x7fa03406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-11T23:02:01.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.897+0000 7fa04cf36700 1 --2- 172.21.15.153:0/165191848 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fa0340692a0 0x7fa03406b750 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa038009b80 tx=0x7fa03800b040 comp rx=0 tx=0).ready entity=mgr.14204 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-11T23:02:01.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:01.901+0000 7fa03dffb700 1 -- 172.21.15.153:0/165191848 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175366 (secure 0 0 0) 0x7fa044058150 con 0x7fa0480fda80 2024-03-11T23:02:02.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:02.133+0000 7fa04f99b700 1 -- 172.21.15.153:0/165191848 --> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa04802b310 con 0x7fa0340692a0 2024-03-11T23:02:02.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:02.135+0000 7fa03dffb700 1 -- 172.21.15.153:0/165191848 <== mgr.14204 v2:172.21.15.153:6800/29938766 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7fa04802b310 con 0x7fa0340692a0 2024-03-11T23:02:02.139 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-03-11T23:02:02.139 INFO:teuthology.orchestra.run.smithi153.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b", 2024-03-11T23:02:02.139 INFO:teuthology.orchestra.run.smithi153.stdout: "in_progress": true, 2024-03-11T23:02:02.139 INFO:teuthology.orchestra.run.smithi153.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-11T23:02:02.139 INFO:teuthology.orchestra.run.smithi153.stdout: "services_complete": [], 2024-03-11T23:02:02.139 INFO:teuthology.orchestra.run.smithi153.stdout: "progress": "0/2 daemons upgraded", 2024-03-11T23:02:02.139 INFO:teuthology.orchestra.run.smithi153.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b image on host smithi163", 2024-03-11T23:02:02.139 INFO:teuthology.orchestra.run.smithi153.stdout: "is_paused": false 2024-03-11T23:02:02.139 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-03-11T23:02:02.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:02.137+0000 7fa04f99b700 1 -- 172.21.15.153:0/165191848 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fa0340692a0 msgr2=0x7fa03406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:02:02.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:02.137+0000 7fa04f99b700 1 --2- 172.21.15.153:0/165191848 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fa0340692a0 0x7fa03406b750 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa038009b80 tx=0x7fa03800b040 comp rx=0 tx=0).stop 2024-03-11T23:02:02.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:02.137+0000 7fa04f99b700 1 -- 172.21.15.153:0/165191848 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa0480fda80 msgr2=0x7fa04810d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-11T23:02:02.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:02.137+0000 7fa04f99b700 1 --2- 172.21.15.153:0/165191848 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa0480fda80 0x7fa04810d8c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa044000c00 tx=0x7fa044009e30 comp rx=0 tx=0).stop 2024-03-11T23:02:02.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:02.137+0000 7fa04f99b700 1 -- 172.21.15.153:0/165191848 shutdown_connections 2024-03-11T23:02:02.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:02.137+0000 7fa04f99b700 1 --2- 172.21.15.153:0/165191848 >> [v2:172.21.15.153:6800/29938766,v1:172.21.15.153:6801/29938766] conn(0x7fa0340692a0 0x7fa03406b750 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:02:02.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:02.137+0000 7fa04f99b700 1 --2- 172.21.15.153:0/165191848 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa0480fda80 0x7fa04810d8c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:02:02.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:02.137+0000 7fa04f99b700 1 --2- 172.21.15.153:0/165191848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa0481003a0 0x7fa04810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-11T23:02:02.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:02.137+0000 7fa04f99b700 1 -- 172.21.15.153:0/165191848 >> 172.21.15.153:0/165191848 conn(0x7fa0480f74e0 msgr2=0x7fa0480f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-11T23:02:02.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:02.138+0000 7fa04f99b700 1 -- 172.21.15.153:0/165191848 shutdown_connections 2024-03-11T23:02:02.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:02:02.138+0000 7fa04f99b700 1 -- 172.21.15.153:0/165191848 wait complete. 2024-03-11T23:02:02.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:02:02 smithi153 ceph-mon[28962]: from='client.14558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:02:02.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:02:02 smithi153 ceph-mon[28962]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:02:02.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:02:02 smithi153 ceph-mon[28962]: from='client.? 172.21.15.153:0/1727973934' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:02:02.737 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:02:02 smithi163 ceph-mon[35584]: from='client.14558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:02:02.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:02:02 smithi163 ceph-mon[35584]: from='client.14562 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:02:02.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:02:02 smithi163 ceph-mon[35584]: from='client.? 172.21.15.153:0/1727973934' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-11T23:02:03.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:02:03 smithi163 ceph-mon[35584]: pgmap v237: 65 pgs: 65 active+clean; 30 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 684 KiB/s rd, 2.5 MiB/s wr, 45 op/s 2024-03-11T23:02:03.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:02:03 smithi163 ceph-mon[35584]: from='client.24351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:02:03.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:02:03 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:02:03.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:02:03 smithi153 ceph-mon[28962]: pgmap v237: 65 pgs: 65 active+clean; 30 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 684 KiB/s rd, 2.5 MiB/s wr, 45 op/s 2024-03-11T23:02:03.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:02:03 smithi153 ceph-mon[28962]: from='client.24351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T23:02:03.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:02:03 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:02:05.737 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:02:05 smithi163 ceph-mon[35584]: pgmap v238: 65 pgs: 65 active+clean; 35 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 48 op/s 2024-03-11T23:02:05.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:02:05 smithi163 ceph-mon[35584]: fsmap cephfs:2 {0=cephfs.smithi153.qxjyxz=up:active,1=cephfs.smithi153.yuzdaj=up:active} 2 up:standby-replay 2024-03-11T23:02:05.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:02:05 smithi153 ceph-mon[28962]: pgmap v238: 65 pgs: 65 active+clean; 35 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 48 op/s 2024-03-11T23:02:05.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:02:05 smithi153 ceph-mon[28962]: fsmap cephfs:2 {0=cephfs.smithi153.qxjyxz=up:active,1=cephfs.smithi153.yuzdaj=up:active} 2 up:standby-replay 2024-03-11T23:02:07.737 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:02:07 smithi163 ceph-mon[35584]: pgmap v239: 65 pgs: 65 active+clean; 42 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.0 MiB/s rd, 3.6 MiB/s wr, 52 op/s 2024-03-11T23:02:07.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:02:07 smithi153 ceph-mon[28962]: pgmap v239: 65 pgs: 65 active+clean; 42 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.0 MiB/s rd, 3.6 MiB/s wr, 52 op/s 2024-03-11T23:02:09.737 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:02:09 smithi163 ceph-mon[35584]: pgmap v240: 65 pgs: 65 active+clean; 42 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.0 MiB/s rd, 3.5 MiB/s wr, 45 op/s 2024-03-11T23:02:09.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:02:09 smithi153 ceph-mon[28962]: pgmap v240: 65 pgs: 65 active+clean; 42 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 2.0 MiB/s rd, 3.5 MiB/s wr, 45 op/s 2024-03-11T23:02:11.738 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:02:11 smithi163 ceph-mon[35584]: pgmap v241: 65 pgs: 65 active+clean; 70 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 3.1 MiB/s rd, 5.6 MiB/s wr, 220 op/s 2024-03-11T23:02:11.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:02:11 smithi153 ceph-mon[28962]: pgmap v241: 65 pgs: 65 active+clean; 70 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 3.1 MiB/s rd, 5.6 MiB/s wr, 220 op/s 2024-03-11T23:02:13.737 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:02:13 smithi163 ceph-mon[35584]: pgmap v242: 65 pgs: 65 active+clean; 90 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.5 MiB/s rd, 5.4 MiB/s wr, 497 op/s 2024-03-11T23:02:13.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:02:13 smithi153 ceph-mon[28962]: pgmap v242: 65 pgs: 65 active+clean; 90 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.5 MiB/s rd, 5.4 MiB/s wr, 497 op/s 2024-03-11T23:02:15.737 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:02:15 smithi163 ceph-mon[35584]: pgmap v243: 65 pgs: 65 active+clean; 118 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 2.8 MiB/s rd, 7.8 MiB/s wr, 781 op/s 2024-03-11T23:02:15.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:02:15 smithi153 ceph-mon[28962]: pgmap v243: 65 pgs: 65 active+clean; 118 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 2.8 MiB/s rd, 7.8 MiB/s wr, 781 op/s 2024-03-11T23:02:17.623 INFO:tasks.workunit.client.1.smithi163.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-03-11T23:02:17.624 INFO:tasks.workunit.client.1.smithi163.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-03-11T23:02:17.624 INFO:tasks.workunit.client.1.smithi163.stderr:+ make 2024-03-11T23:02:17.737 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:02:17 smithi163 ceph-mon[35584]: pgmap v244: 65 pgs: 65 active+clean; 139 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.3 MiB/s rd, 9.2 MiB/s wr, 1.06k op/s 2024-03-11T23:02:17.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:02:17 smithi153 ceph-mon[28962]: pgmap v244: 65 pgs: 65 active+clean; 139 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.3 MiB/s rd, 9.2 MiB/s wr, 1.06k op/s 2024-03-11T23:02:18.104 INFO:tasks.workunit.client.1.smithi163.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-03-11T23:02:18.487 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:02:18 smithi163 ceph-mon[35584]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:02:18.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:02:18 smithi153 ceph-mon[28962]: from='mgr.14204 172.21.15.153:0/1626414986' entity='mgr.smithi153.nyocep' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T23:02:19.737 INFO:journalctl@ceph.mon.smithi163.smithi163.stdout:Mar 11 23:02:19 smithi163 ceph-mon[35584]: pgmap v245: 65 pgs: 65 active+clean; 149 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 9.5 MiB/s wr, 1.17k op/s 2024-03-11T23:02:19.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:02:19 smithi153 ceph-mon[28962]: pgmap v245: 65 pgs: 65 active+clean; 149 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 9.5 MiB/s wr, 1.17k op/s 2024-03-11T23:02:41.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:41.153+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:41.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:41.153+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:41.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:41.153+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:42.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:42.195+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:42.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:42.195+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:42.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:42.195+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:43.513 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:43.232+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:43.513 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:43.232+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:43.513 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:43.232+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:43.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:43.525+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:43.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:43.525+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:43.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:43.525+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:44.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:44.257+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:44.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:44.257+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:44.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:44.258+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:45.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:44.552+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:45.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:44.552+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:45.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:44.552+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:45.523 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:45.226+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:45.524 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:45.226+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:45.524 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:45.226+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:45.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:45.519+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:45.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:45.519+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:45.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:45.519+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:45.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:45.656+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:45.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:45.656+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:45.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:45.656+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:46.498 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:46.183+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:46.498 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:46.183+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:46.498 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:46.183+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:46.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:46.660+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:46.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:46.660+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:46.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:46.660+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:46.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:46.494+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:46.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:46.494+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:46.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:46.494+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:47.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:47.158+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:47.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:47.158+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:47.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:47.158+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:48.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:47.673+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:48.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:47.673+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:48.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:47.673+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:48.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:47.527+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:48.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:47.527+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:48.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:47.527+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:48.483 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:48.117+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:48.483 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:48.117+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:48.484 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:48.117+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:48.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:48.630+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:48.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:48.630+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:48.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:48.630+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:48.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:48.479+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:48.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:48.479+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:48.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:48.479+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:49.458 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:49.161+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:49.458 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:49.161+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:49.458 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:49.161+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:49.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:49.679+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:49.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:49.679+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:49.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:49.679+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:49.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:49.521+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:49.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:49.521+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:49.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:49.521+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:50.511 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:50.178+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:50.511 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:50.178+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:50.511 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:50.178+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:50.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:50.507+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:50.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:50.507+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:50.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:50.507+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:50.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:50.670+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:50.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:50.670+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:50.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:50.670+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:51.468 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:51.175+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:51.468 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:51.175+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:51.468 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:51.175+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:51.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:51.686+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:51.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:51.686+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:51.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:51.686+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:51.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:51.463+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:51.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:51.464+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:51.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:51.464+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:52.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:52.441+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:52.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:52.441+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:52.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:52.441+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:52.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:52.219+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:52.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:52.219+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:52.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:52.219+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:53.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:52.669+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:53.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:52.669+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:53.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:52.669+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:53.453 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:53.193+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:53.453 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:53.193+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:53.453 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:53.193+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:53.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:53.686+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:53.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:53.686+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:53.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:53.686+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:53.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:53.449+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:53.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:53.449+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:53.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:53.449+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:54.421 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:54.146+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:54.421 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:54.146+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:54.421 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:54.146+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:54.690 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:54.416+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:54.690 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:54.416+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:54.690 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:54.416+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:55.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:54.685+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:55.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:54.686+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:55.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:54.686+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:55.431 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:55.127+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:55.431 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:55.127+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:55.431 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:55.127+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:55.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:55.670+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:55.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:55.670+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:55.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:55.670+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:55.782 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:55.426+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:55.782 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:55.426+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:55.782 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:55.426+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:56.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:56.409+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:56.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:56.409+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:56.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:56.409+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:56.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:56.165+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:56.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:56.165+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:56.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:56.165+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:57.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:56.624+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:57.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:56.624+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:57.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:56.624+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:57.457 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:57.117+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:57.458 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:57.117+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:57.458 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:57.117+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:57.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:57.673+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:57.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:57.673+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:57.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:57.673+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:57.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:57.453+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:57.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:57.453+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:57.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:57.453+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:58.423 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:58.161+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:58.423 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:58.161+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:58.423 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:58.161+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:58.696 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:58.419+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:58.696 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:58.419+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:58.697 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:58.419+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:59.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:58.692+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:59.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:58.692+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:59.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:58.692+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:59.458 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:59.374+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:59.458 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:59.374+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:59.458 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:02:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:02:59.374+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:02:59.458 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:59.148+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:59.458 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:59.148+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:59.458 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:02:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:02:59.148+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:02:59.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:59.695+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:59.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:59.695+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:02:59.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:02:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:02:59.695+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:00.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:00.391+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:00.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:00.391+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:00.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:00.391+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:00.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:00.152+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:00.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:00.153+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:00.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:00.153+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:01.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:00.724+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:01.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:00.724+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:01.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:00.725+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:01.415 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:01.125+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:01.415 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:01.125+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:01.415 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:01.125+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:01.703 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:01.411+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:01.704 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:01.411+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:01.704 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:01.411+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:02.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:01.699+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:02.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:01.699+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:02.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:01.699+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:02.441 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:02.113+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:02.442 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:02.113+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:02.442 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:02.113+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:02.705 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:02.437+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:02.705 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:02.437+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:02.705 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:02.437+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:03.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:02.701+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:03.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:02.701+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:03.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:02.701+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:03.475 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:03.125+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:03.475 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:03.125+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:03.475 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:03.125+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:03.733 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:03.471+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:03.733 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:03.471+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:03.733 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:03.471+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:04.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:03.728+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:04.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:03.728+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:04.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:03.728+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:04.346 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:04.091+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:04.346 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:04.091+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:04.346 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:04.091+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:04.743 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:04.426+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:04.744 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:04.426+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:04.744 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:04.426+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:05.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:04.739+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:05.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:04.739+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:05.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:04.739+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:05.448 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:05.127+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:05.448 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:05.127+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:05.448 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:05.127+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:05.761 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:05.443+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:05.761 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:05.443+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:05.761 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:05.443+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:06.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:05.757+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:06.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:05.757+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:06.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:05.757+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:06.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:06.169+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:06.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:06.169+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:06.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:06.169+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:06.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:06.417+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:06.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:06.417+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:06.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:06.417+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:07.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:06.770+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:07.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:06.770+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:07.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:06.770+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:07.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:07.373+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:07.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:07.373+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:07.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:07.373+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:07.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:07.213+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:07.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:07.213+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:07.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:07.213+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:08.183 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:07.806+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:08.184 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:07.806+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:08.184 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:07.806+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:08.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:08.358+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:08.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:08.358+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:08.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:08.358+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:08.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:08.179+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:08.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:08.179+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:08.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:08.179+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:09.198 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:08.849+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:09.198 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:08.849+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:09.198 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:08.849+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:09.461 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:09.337+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:09.462 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:09.337+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:09.462 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:09.337+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:09.462 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:09.194+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:09.462 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:09.194+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:09.462 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:09.194+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:10.202 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:09.812+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:10.202 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:09.812+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:10.202 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:09.812+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:10.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:10.295+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:10.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:10.295+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:10.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:10.295+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:10.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:10.198+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:10.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:10.198+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:10.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:10.198+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:11.176 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:10.852+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:11.176 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:10.852+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:11.176 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:10.852+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:11.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:11.256+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:11.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:11.256+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:11.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:11.256+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:11.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:11.172+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:11.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:11.172+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:11.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:11.172+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:12.189 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:11.858+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:12.189 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:11.858+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:12.189 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:11.858+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:12.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:12.223+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:12.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:12.223+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:12.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:12.223+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:12.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:12.185+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:12.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:12.185+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:12.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:12.185+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:13.185 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:12.861+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:13.185 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:12.861+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:13.185 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:12.861+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:13.515 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:13.243+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:13.515 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:13.243+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:13.516 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:13.243+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:13.516 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:13.181+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:13.516 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:13.181+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:13.516 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:13.181+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:14.150 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:13.855+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:14.150 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:13.855+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:14.150 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:13.855+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:14.459 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:14.262+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:14.459 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:14.262+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:14.459 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:14.262+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:14.459 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:14.146+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:14.460 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:14.146+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:14.460 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:14.146+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:15.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:15.129+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:15.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:15.129+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:15.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:15.129+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:15.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:14.881+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:15.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:14.881+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:15.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:14.881+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:15.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:15.300+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:15.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:15.300+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:15.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:15.300+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:16.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:15.908+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:16.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:15.908+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:16.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:15.908+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:16.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:16.125+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:16.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:16.125+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:16.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:16.125+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:16.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:16.298+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:16.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:16.298+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:16.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:16.298+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:17.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:16.951+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:17.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:16.951+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:17.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:16.951+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:17.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:17.087+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:17.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:17.087+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:17.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:17.087+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:17.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:17.344+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:17.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:17.344+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:17.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:17.344+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:18.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:18.001+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:18.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:18.001+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:18.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:18.001+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:18.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:18.058+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:18.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:18.058+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:18.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:18.058+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:18.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:18.381+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:18.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:18.381+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:18.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:18.381+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:19.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:19.100+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:19.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:19.100+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:19.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:19.100+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:19.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:19.022+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:19.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:19.022+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:19.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:19.022+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:19.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:19.409+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:19.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:19.409+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:19.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:19.409+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:20.425 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:20.100+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:20.425 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:20.100+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:20.425 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:20.100+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:20.425 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:20.056+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:20.425 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:20.056+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:20.425 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:20.056+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:20.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:20.421+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:20.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:20.421+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:20.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:20.421+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:21.468 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:21.087+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:21.468 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:21.087+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:21.468 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:21.087+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:21.468 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:21.051+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:21.468 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:21.051+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:21.469 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:21.051+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:21.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:21.464+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:21.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:21.464+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:21.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:21.464+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:22.449 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:22.074+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:22.450 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:22.074+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:22.450 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:22.074+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:22.450 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:22.040+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:22.450 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:22.040+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:22.450 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:22.040+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:22.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:22.445+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:22.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:22.445+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:22.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:22.445+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:23.474 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:23.062+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:23.474 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:23.062+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:23.474 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:23.062+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:23.475 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:23.083+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:23.475 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:23.083+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:23.475 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:23.083+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:23.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:23.470+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:23.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:23.470+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:23.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:23.470+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:24.349 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:24.078+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:24.349 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:24.078+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:24.349 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:24.078+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:24.350 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:24.047+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:24.350 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:24.047+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:24.350 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:24.047+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:24.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:24.475+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:24.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:24.475+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:24.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:24.475+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:25.510 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:25.070+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:25.510 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:25.070+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:25.510 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:25.070+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:25.511 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:25.075+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:25.511 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:25.075+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:25.511 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:25.075+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:25.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:25.506+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:25.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:25.506+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:25.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:25.506+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:26.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:26.081+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:26.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:26.081+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:26.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:26.082+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:26.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:26.100+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:26.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:26.100+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:26.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:26.100+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:27.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:26.552+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:27.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:26.552+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:27.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:26.552+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:27.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:27.069+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:27.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:27.069+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:27.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:27.069+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:27.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:27.033+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:27.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:27.033+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:27.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:27.033+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:28.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:27.530+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:28.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:27.530+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:28.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:27.530+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:28.502 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:28.052+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:28.502 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:28.052+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:28.502 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:28.052+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:28.503 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:28.078+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:28.503 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:28.078+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:28.503 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:28.078+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:28.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:28.498+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:28.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:28.498+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:28.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:28.498+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:29.340 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:29.072+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:29.340 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:29.072+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:29.340 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:29.072+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:29.341 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:29.031+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:29.341 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:29.031+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:29.341 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:29.031+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:29.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:29.460+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:29.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:29.460+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:29.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:29.460+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:30.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:30.009+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:30.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:30.009+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:30.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:30.009+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:30.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:30.035+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:30.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:30.035+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:30.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:30.035+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:30.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:30.454+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:30.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:30.454+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:30.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:30.454+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:31.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:31.004+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:31.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:31.004+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:31.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:31.004+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:31.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:31.049+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:31.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:31.049+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:31.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:31.049+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:31.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:31.502+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:31.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:31.502+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:31.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:31.502+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:32.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:32.020+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:32.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:32.020+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:32.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:32.020+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:32.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:32.028+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:32.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:32.028+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:32.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:32.028+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:32.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:32.509+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:32.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:32.509+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:32.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:32.509+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:33.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:32.992+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:33.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:32.992+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:33.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:32.992+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:33.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:33.022+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:33.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:33.022+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:33.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:33.022+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:33.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:33.543+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:33.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:33.543+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:33.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:33.543+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:34.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:34.032+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:34.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:34.032+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:34.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:34.032+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:34.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:33.995+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:34.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:33.995+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:34.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:33.995+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:34.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:34.568+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:34.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:34.568+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:34.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:34.568+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:35.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:35.014+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:35.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:35.014+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:35.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:35.014+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:35.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:35.024+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:35.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:35.024+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:35.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:35.024+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:36.002 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:35.548+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:36.002 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:35.548+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:36.002 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:35.548+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:36.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:36.014+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:36.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:36.014+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:36.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:36.014+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:36.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:35.997+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:36.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:35.997+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:36.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:35.997+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:36.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:36.525+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:36.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:36.525+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:36.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:36.525+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:37.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:36.992+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:37.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:36.992+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:37.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:36.992+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:37.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:36.989+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:37.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:36.989+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:37.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:36.989+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:37.968 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:37.539+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:37.968 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:37.539+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:37.968 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:37.539+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:38.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:37.976+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:38.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:37.976+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:38.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:37.976+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:38.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:37.963+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:38.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:37.963+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:38.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:37.963+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:38.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:38.489+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:38.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:38.489+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:38.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:38.489+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:39.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:38.992+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:39.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:38.992+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:39.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:38.992+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:39.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:38.956+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:39.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:38.956+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:39.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:38.956+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:39.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:39.513+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:39.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:39.513+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:39.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:39.513+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:40.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:40.025+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:40.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:40.025+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:40.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:40.025+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:40.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:40.002+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:40.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:40.002+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:40.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:40.002+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:40.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:40.517+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:40.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:40.517+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:40.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:40.517+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:41.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:40.964+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:41.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:40.964+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:41.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:40.964+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:41.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:41.041+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:41.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:41.041+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:41.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:41.041+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:41.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:41.468+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:41.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:41.468+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:41.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:41.468+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:42.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:42.070+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:42.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:42.070+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:42.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:42.070+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:42.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:41.976+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:42.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:41.976+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:42.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:41.976+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:42.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:42.440+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:42.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:42.440+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:42.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:42.440+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:43.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:42.995+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:43.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:42.995+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:43.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:42.995+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:43.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:43.116+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:43.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:43.116+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:43.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:43.116+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:43.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:43.461+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:43.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:43.461+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:43.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:43.461+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:44.350 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:44.069+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:44.351 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:44.069+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:44.351 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:44.069+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:44.351 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:44.037+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:44.351 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:44.037+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:44.351 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:44.037+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:44.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:44.496+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:44.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:44.496+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:44.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:44.496+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:45.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:45.061+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:45.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:45.061+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:45.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:45.061+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:45.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:45.008+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:45.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:45.008+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:45.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:45.008+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:45.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:45.456+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:45.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:45.456+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:45.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:45.456+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:46.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:46.072+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:46.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:46.072+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:46.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:46.072+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:46.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:46.021+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:46.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:46.021+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:46.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:46.021+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:46.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:46.503+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:46.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:46.503+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:46.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:46.503+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:47.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:46.987+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:47.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:46.988+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:47.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:46.988+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:47.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:47.062+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:47.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:47.062+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:47.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:47.062+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:47.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:47.476+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:47.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:47.476+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:47.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:47.476+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:48.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:48.028+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:48.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:48.028+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:48.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:48.028+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:48.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:47.984+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:48.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:47.984+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:48.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:47.984+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:48.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:48.460+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:48.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:48.460+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:48.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:48.460+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:49.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:48.979+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:49.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:48.979+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:49.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:48.979+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:49.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:49.025+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:49.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:49.026+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:49.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:49.026+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:49.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:49.418+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:49.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:49.418+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:49.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:49.418+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:50.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:49.991+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:50.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:49.991+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:50.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:49.991+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:50.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:50.005+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:50.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:50.005+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:50.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:50.005+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:50.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:50.395+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:50.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:50.395+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:50.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:50.395+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:51.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:50.958+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:51.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:50.958+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:51.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:50.958+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:51.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:50.979+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:51.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:50.979+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:51.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:50.979+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:51.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:51.436+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:51.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:51.436+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:51.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:51.436+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:52.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:51.938+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:52.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:51.938+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:52.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:51.938+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:52.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:51.999+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:52.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:51.999+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:52.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:51.999+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:52.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:52.475+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:52.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:52.475+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:52.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:52.475+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:53.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:52.940+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:53.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:52.940+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:53.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:52.940+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:53.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:53.035+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:53.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:53.035+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:53.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:53.035+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:53.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:53.429+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:53.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:53.429+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:53.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:53.429+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:54.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:53.932+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:54.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:53.932+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:54.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:53.932+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:54.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:54.009+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:54.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:54.009+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:54.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:54.009+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:54.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:54.395+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:54.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:54.395+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:54.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:54.395+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:55.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:55.035+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:55.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:55.035+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:55.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:55.035+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:55.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:54.947+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:55.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:54.947+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:55.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:54.947+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:55.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:55.368+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:55.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:55.368+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:55.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:55.368+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:56.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:55.905+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:56.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:55.905+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:56.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:55.905+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:56.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:56.017+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:56.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:56.017+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:56.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:56.017+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:56.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:56.324+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:56.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:56.324+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:56.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:56.324+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:57.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:57.013+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:57.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:57.013+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:57.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:57.013+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:57.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:56.944+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:57.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:56.944+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:57.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:56.944+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:57.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:57.299+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:57.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:57.299+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:57.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:57.299+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:58.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:58.043+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:58.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:58.043+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:58.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:58.043+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:58.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:57.910+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:58.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:57.910+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:58.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:57.910+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:58.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:58.308+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:58.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:58.308+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:58.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:58.309+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:59.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:58.897+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:59.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:58.897+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:59.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:58.897+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:03:59.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:59.025+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:59.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:59.025+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:59.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:59.025+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:03:59.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:59.276+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:59.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:59.276+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:03:59.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:03:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:03:59.276+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:00.239 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:59.879+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:00.239 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:59.879+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:00.239 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:03:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:03:59.879+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:00.239 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:59.988+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:00.239 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:59.988+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:00.239 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:03:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:03:59.988+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:00.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:00.235+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:00.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:00.235+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:00.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:00.235+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:01.226 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:01.000+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:01.226 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:01.000+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:01.226 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:01.000+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:01.226 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:00.899+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:01.226 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:00.899+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:01.226 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:00.899+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:01.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:01.221+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:01.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:01.221+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:01.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:01.221+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:01.957 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:01.859+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:01.957 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:01.859+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:01.957 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:01.859+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:02.214 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:01.953+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:02.214 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:01.953+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:02.214 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:01.953+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:02.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:02.224+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:02.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:02.224+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:02.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:02.224+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:03.202 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:02.816+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:03.202 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:02.816+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:03.202 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:02.816+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:03.202 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:03.001+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:03.202 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:03.001+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:03.203 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:03.001+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:03.517 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:03.198+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:03.518 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:03.198+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:03.518 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:03.198+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:04.054 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:03.793+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:04.054 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:03.793+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:04.054 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:03.793+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:04.354 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:04.050+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:04.354 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:04.050+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:04.354 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:04.050+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:04.354 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:04.233+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:04.354 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:04.233+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:04.354 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:04.233+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:05.101 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:04.815+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:05.101 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:04.815+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:05.101 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:04.815+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:05.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:05.097+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:05.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:05.097+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:05.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:05.097+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:05.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:05.240+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:05.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:05.240+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:05.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:05.240+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:06.133 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:05.861+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:06.133 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:05.861+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:06.133 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:05.861+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:06.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:06.129+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:06.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:06.129+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:06.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:06.129+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:06.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:06.286+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:06.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:06.286+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:06.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:06.286+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:07.260 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:06.908+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:07.260 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:06.908+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:07.260 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:06.908+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:07.261 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:07.123+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:07.261 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:07.123+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:07.261 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:07.123+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:07.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:07.256+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:07.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:07.256+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:07.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:07.256+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:08.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:08.158+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:08.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:08.158+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:08.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:08.158+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:08.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:07.915+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:08.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:07.915+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:08.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:07.915+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:08.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:08.296+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:08.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:08.296+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:08.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:08.296+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:09.262 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:08.886+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:09.262 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:08.886+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:09.262 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:08.886+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:09.262 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:09.125+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:09.263 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:09.125+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:09.263 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:09.125+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:09.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:09.258+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:09.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:09.258+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:09.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:09.258+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:10.147 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:09.844+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:10.147 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:09.844+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:10.147 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:09.844+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:10.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:10.143+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:10.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:10.143+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:10.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:10.143+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:10.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:10.212+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:10.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:10.213+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:10.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:10.213+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:11.182 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:10.858+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:11.182 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:10.858+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:11.182 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:10.858+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:11.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:11.178+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:11.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:11.178+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:11.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:11.178+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:11.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:11.182+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:11.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:11.182+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:11.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:11.182+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:12.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:11.814+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:12.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:11.814+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:12.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:11.814+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:12.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:12.132+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:12.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:12.132+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:12.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:12.132+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:12.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:12.179+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:12.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:12.179+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:12.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:12.179+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:13.148 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:12.835+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:13.148 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:12.835+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:13.148 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:12.835+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:13.518 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:13.186+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:13.518 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:13.186+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:13.518 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:13.186+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:13.518 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:13.144+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:13.518 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:13.144+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:13.518 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:13.144+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:14.181 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:13.794+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:14.181 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:13.794+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:14.181 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:13.794+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:14.466 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:14.235+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:14.467 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:14.235+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:14.467 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:14.235+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:14.467 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:14.177+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:14.467 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:14.177+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:14.467 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:14.177+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:15.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:14.746+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:15.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:14.746+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:15.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:14.746+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:15.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:15.201+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:15.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:15.201+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:15.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:15.201+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:15.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:15.196+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:15.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:15.196+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:15.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:15.196+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:16.161 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:15.787+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:16.161 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:15.787+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:16.161 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:15.787+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:16.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:16.179+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:16.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:16.179+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:16.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:16.179+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:16.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:16.157+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:16.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:16.157+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:16.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:16.157+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:17.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:16.765+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:17.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:16.765+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:17.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:16.765+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:17.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:17.145+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:17.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:17.145+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:17.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:17.145+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:17.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:17.203+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:17.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:17.203+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:17.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:17.203+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:18.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:17.751+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:18.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:17.751+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:18.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:17.751+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:18.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:18.107+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:18.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:18.107+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:18.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:18.107+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:18.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:18.251+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:18.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:18.251+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:18.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:18.251+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:19.083 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:18.776+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:19.083 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:18.776+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:19.083 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:18.776+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:19.355 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:19.255+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:19.356 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:19.255+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:19.356 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:19.255+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:19.356 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:19.079+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:19.356 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:19.079+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:19.356 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:19.079+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:20.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:19.753+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:20.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:19.753+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:20.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:19.753+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:20.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:20.269+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:20.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:20.269+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:20.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:20.269+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:20.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:20.042+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:20.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:20.042+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:20.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:20.042+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:21.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:20.731+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:21.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:20.731+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:21.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:20.731+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:21.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:21.088+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:21.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:21.088+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:21.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:21.088+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:21.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:21.310+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:21.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:21.310+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:21.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:21.310+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:22.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:21.720+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:22.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:21.720+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:22.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:21.720+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:22.314 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:22.043+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:22.315 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:22.043+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:22.315 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:22.043+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:22.726 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:22.310+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:22.726 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:22.310+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:22.726 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:22.310+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:23.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:22.722+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:23.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:22.722+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:23.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:22.722+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:23.354 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:23.034+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:23.354 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:23.034+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:23.354 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:23.034+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:23.718 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:23.350+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:23.718 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:23.350+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:23.718 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:23.350+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:24.015 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:23.745+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:24.016 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:23.745+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:24.016 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:23.745+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:24.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:24.012+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:24.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:24.012+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:24.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:24.012+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:24.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:04:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:04:24.340+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 3 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:04:24.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:24.331+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:24.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:24.331+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:24.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:24.331+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:25.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:24.973+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:25.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:24.973+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:25.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:24.973+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:25.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:24.786+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:25.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:24.786+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:25.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:24.786+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:25.744 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:25.314+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:25.744 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:25.314+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:25.744 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:25.314+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:26.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:25.984+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:26.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:25.984+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:26.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:25.984+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:26.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:25.741+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:26.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:25.741+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:26.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:25.741+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:26.718 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:26.357+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:26.719 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:26.357+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:26.719 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:26.357+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:27.006 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:26.715+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:27.006 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:26.715+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:27.007 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:26.715+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:27.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:27.002+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:27.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:27.002+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:27.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:27.002+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:27.729 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:27.398+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:27.729 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:27.398+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:27.729 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:27.398+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:28.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:27.726+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:28.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:27.726+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:28.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:27.726+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:28.361 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:28.032+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:28.361 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:28.032+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:28.361 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:28.032+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:28.687 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:28.358+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:28.687 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:28.358+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:28.687 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:28.358+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:29.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:28.683+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:29.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:28.683+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:29.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:28.683+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:29.344 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:29.065+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:29.345 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:29.065+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:29.345 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:29.065+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:29.656 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:04:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:04:29.340+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 18 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:04:29.656 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:29.387+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:29.656 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:29.387+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:29.656 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:29.387+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:30.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:29.652+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:30.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:29.652+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:30.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:29.652+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:30.356 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:30.051+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:30.356 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:30.051+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:30.356 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:30.051+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:30.663 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:30.352+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:30.663 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:30.352+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:30.663 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:30.352+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:31.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:30.659+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:31.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:30.659+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:31.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:30.659+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:31.319 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:31.026+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:31.319 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:31.026+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:31.319 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:31.026+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:31.630 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:31.315+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:31.630 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:31.315+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:31.630 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:31.315+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:32.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:31.635+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:32.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:31.635+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:32.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:31.635+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:32.318 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:32.061+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:32.353 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:32.061+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:32.353 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:32.061+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:32.661 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:32.314+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:32.661 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:32.314+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:32.661 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:32.314+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:32.934 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:32.657+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:32.934 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:32.657+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:32.934 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:32.657+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:33.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:33.050+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:33.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:33.050+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:33.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:33.050+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:33.620 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:33.286+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:33.621 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:33.286+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:33.621 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:33.286+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:34.020 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:33.616+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:34.020 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:33.616+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:34.020 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:33.616+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:34.273 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:34.015+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:34.273 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:34.015+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:34.273 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:34.015+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:34.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:04:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:04:34.341+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 34 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:04:34.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:34.268+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:34.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:34.268+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:34.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:34.268+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:35.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:34.610+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:35.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:34.610+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:35.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:34.610+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:35.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:35.024+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:35.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:35.024+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:35.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:35.024+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:35.567 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:35.307+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:35.567 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:35.307+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:35.567 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:35.307+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:36.007 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:35.563+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:36.008 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:35.563+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:36.008 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:35.563+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:36.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:36.015+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:36.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:36.015+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:36.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:36.015+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:36.558 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:36.281+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:36.558 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:36.281+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:36.558 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:36.281+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:36.970 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:36.554+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:36.970 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:36.554+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:36.970 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:36.554+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:37.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:36.966+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:37.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:36.966+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:37.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:36.966+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:37.570 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:37.305+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:37.570 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:37.305+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:37.570 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:37.305+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:37.986 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:37.565+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:37.986 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:37.565+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:37.986 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:37.565+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:38.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:37.981+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:38.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:37.981+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:38.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:37.981+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:38.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:38.341+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:38.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:38.341+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:38.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:38.341+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:38.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:38.528+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:38.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:38.528+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:38.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:38.528+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:39.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:38.970+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:39.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:38.970+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:39.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:38.970+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:39.550 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:04:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:04:39.342+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 49 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:04:39.550 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:39.349+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:39.551 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:39.349+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:39.551 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:39.349+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:39.977 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:39.546+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:39.977 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:39.546+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:39.977 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:39.546+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:40.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:39.973+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:40.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:39.973+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:40.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:39.973+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:40.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:40.358+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:40.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:40.358+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:40.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:40.358+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:40.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:40.547+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:40.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:40.547+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:40.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:40.547+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:41.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:40.933+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:41.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:40.933+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:41.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:40.933+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:41.631 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:41.564+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:41.631 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:41.564+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:41.631 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:41.564+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:41.631 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:41.327+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:41.631 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:41.327+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:41.631 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:41.327+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:42.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:41.966+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:42.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:41.966+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:42.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:41.966+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:42.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:42.377+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:42.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:42.377+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:42.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:42.377+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:42.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:42.548+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:42.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:42.548+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:42.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:42.548+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:43.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:42.937+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:43.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:42.937+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:43.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:42.937+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:43.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:43.329+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:43.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:43.329+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:43.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:43.329+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:43.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:43.533+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:43.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:43.533+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:43.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:43.533+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:44.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:43.965+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:44.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:43.965+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:44.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:43.965+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:44.560 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:04:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:04:44.342+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 84 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:04:44.560 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:44.297+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:44.560 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:44.297+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:44.561 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:44.297+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:44.990 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:44.556+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:44.990 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:44.556+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:44.990 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:44.556+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:45.271 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:44.986+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:45.271 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:44.986+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:45.271 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:44.986+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:45.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:45.267+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:45.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:45.267+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:45.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:45.267+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:45.993 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:45.565+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:45.993 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:45.565+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:45.993 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:45.565+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:46.250 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:45.989+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:46.250 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:45.989+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:46.251 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:45.989+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:46.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:46.246+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:46.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:46.246+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:46.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:46.246+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:47.014 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:46.608+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:47.014 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:46.608+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:47.014 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:46.608+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:47.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:47.010+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:47.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:47.010+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:47.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:47.010+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:47.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:47.241+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:47.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:47.241+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:47.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:47.241+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:48.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:47.624+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:48.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:47.624+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:48.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:47.624+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:48.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:48.052+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:48.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:48.052+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:48.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:48.052+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:48.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:48.251+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:48.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:48.251+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:48.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:48.251+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:49.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:48.574+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:49.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:48.574+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:49.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:48.574+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:49.347 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:49.231+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:49.347 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:49.231+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:49.347 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:49.231+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:49.347 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:49.047+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:49.348 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:49.047+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:49.348 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:49.047+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:49.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:04:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:04:49.343+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 112 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:04:49.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:49.534+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:49.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:49.534+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:49.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:49.534+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:50.515 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:50.270+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:50.515 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:50.270+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:50.515 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:50.270+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:50.516 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:50.056+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:50.516 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:50.056+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:50.516 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:50.056+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:50.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:50.511+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:50.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:50.511+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:50.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:50.511+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:51.523 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:51.276+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:51.523 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:51.276+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:51.523 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:51.276+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:51.524 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:51.065+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:51.524 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:51.065+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:51.524 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:51.065+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:51.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:51.519+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:51.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:51.519+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:51.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:51.519+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:52.486 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:52.289+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:52.486 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:52.289+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:52.487 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:52.289+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:52.487 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:52.073+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:52.487 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:52.073+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:52.487 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:52.073+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:52.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:52.482+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:52.833 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:52.482+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:52.833 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:52.482+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:53.325 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:53.036+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:53.325 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:53.036+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:53.325 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:53.037+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:53.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:53.321+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:53.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:53.321+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:53.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:53.321+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:53.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:53.525+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:53.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:53.525+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:53.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:53.525+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:54.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:54.004+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:54.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:54.004+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:54.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:54.004+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:54.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:04:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:04:54.343+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 139 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:04:54.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:54.318+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:54.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:54.318+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:54.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:54.318+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:54.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:54.523+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:54.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:54.523+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:54.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:54.523+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:55.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:54.973+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:55.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:54.973+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:55.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:54.973+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:55.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:55.300+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:55.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:55.300+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:55.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:55.300+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:55.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:55.502+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:55.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:55.502+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:55.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:55.502+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:56.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:55.955+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:56.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:55.955+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:56.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:55.955+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:56.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:56.283+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:56.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:56.283+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:56.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:56.283+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:56.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:56.527+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:56.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:56.527+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:56.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:56.527+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:57.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:56.958+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:57.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:56.958+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:57.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:56.958+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:57.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:57.331+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:57.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:57.331+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:57.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:57.332+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:57.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:57.540+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:57.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:57.540+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:57.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:57.540+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:58.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:57.953+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:58.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:57.953+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:58.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:57.953+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:58.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:58.288+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:58.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:58.288+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:58.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:58.288+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:58.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:58.511+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:58.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:58.511+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:58.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:58.511+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:59.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:58.912+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:59.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:58.912+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:59.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:58.912+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:04:59.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:04:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:04:59.344+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 171 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:04:59.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:59.325+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:59.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:59.325+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:59.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:04:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:04:59.325+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:04:59.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:59.532+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:59.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:59.532+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:04:59.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:04:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:04:59.532+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:00.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:59.890+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:00.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:59.890+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:00.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:04:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:04:59.890+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:00.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:00.361+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:00.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:00.361+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:00.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:00.361+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:00.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:00.564+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:00.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:00.564+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:00.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:00.564+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:01.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:00.857+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:01.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:00.857+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:01.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:00.857+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:01.632 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:01.548+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:01.632 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:01.548+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:01.632 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:01.548+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:01.633 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:01.359+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:01.633 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:01.359+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:01.633 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:01.359+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:02.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:01.856+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:02.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:01.856+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:02.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:01.856+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:02.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:02.507+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:02.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:02.507+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:02.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:02.507+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:02.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:02.398+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:02.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:02.398+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:02.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:02.398+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:03.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:02.827+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:03.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:02.827+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:03.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:02.827+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:03.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:03.438+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:03.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:03.438+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:03.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:03.438+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:03.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:03.502+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:03.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:03.502+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:03.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:03.502+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:04.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:03.808+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:04.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:03.808+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:04.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:03.808+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:04.771 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:05:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:05:04.344+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 200 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:05:04.772 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:04.456+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:04.772 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:04.456+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:04.772 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:04.456+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:04.772 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:04.500+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:04.772 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:04.500+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:04.772 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:04.500+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:05.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:04.767+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:05.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:04.767+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:05.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:04.767+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:05.771 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:05.409+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:05.771 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:05.409+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:05.772 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:05.409+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:05.772 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:05.455+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:05.772 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:05.455+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:05.772 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:05.455+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:06.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:05.767+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:06.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:05.767+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:06.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:05.767+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:06.727 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:06.427+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:06.727 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:06.427+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:06.727 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:06.427+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:06.728 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:06.445+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:06.728 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:06.445+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:06.728 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:06.445+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:07.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:06.754+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:07.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:06.754+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:07.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:06.754+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:07.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:07.384+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:07.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:07.384+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:07.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:07.384+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:07.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:07.406+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:07.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:07.406+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:07.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:07.406+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:08.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:07.780+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:08.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:07.780+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:08.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:07.780+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:08.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:08.411+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:08.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:08.411+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:08.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:08.411+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:08.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:08.417+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:08.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:08.417+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:08.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:08.417+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:09.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:08.792+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:09.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:08.792+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:09.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:08.792+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:09.751 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:05:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:05:09.344+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 229 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:05:09.752 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:09.375+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:09.752 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:09.376+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:09.752 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:09.376+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:09.752 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:09.405+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:09.752 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:09.405+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:09.752 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:09.405+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:10.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:09.747+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:10.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:09.747+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:10.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:09.747+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:10.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:10.362+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:10.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:10.362+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:10.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:10.362+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:10.779 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:10.378+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:10.779 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:10.378+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:10.779 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:10.378+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:11.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:10.774+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:11.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:10.774+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:11.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:10.774+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:11.633 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:11.328+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:11.633 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:11.328+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:11.633 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:11.328+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:11.633 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:11.341+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:11.633 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:11.341+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:11.633 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:11.341+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:12.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:11.776+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:12.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:11.776+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:12.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:11.776+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:12.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:12.330+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:12.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:12.330+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:12.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:12.330+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:12.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:12.364+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:12.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:12.364+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:12.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:12.364+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:13.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:12.785+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:13.330 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:12.785+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:13.330 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:12.785+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:13.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:13.342+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:13.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:13.342+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:13.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:13.342+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:13.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:13.324+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:13.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:13.324+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:13.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:13.324+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:14.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:13.833+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:14.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:13.833+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:14.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:13.833+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:14.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:05:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:05:14.345+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 256 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:05:14.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:14.337+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:14.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:14.337+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:14.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:14.337+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:14.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:14.294+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:14.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:14.294+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:14.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:14.294+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:15.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:14.803+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:15.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:14.803+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:15.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:14.803+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:15.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:15.327+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:15.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:15.327+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:15.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:15.327+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:15.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:15.312+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:15.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:15.312+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:15.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:15.312+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:16.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:15.779+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:16.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:15.779+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:16.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:15.779+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:16.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:16.330+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:16.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:16.330+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:16.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:16.330+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:16.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:16.326+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:16.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:16.326+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:16.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:16.326+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:17.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:16.826+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:17.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:16.826+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:17.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:16.826+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:17.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:17.329+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:17.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:17.329+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:17.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:17.329+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:17.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:17.340+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:17.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:17.340+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:17.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:17.340+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:18.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:17.857+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:18.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:17.857+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:18.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:17.857+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:18.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:18.336+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:18.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:18.336+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:18.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:18.336+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:18.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:18.384+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:18.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:18.384+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:18.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:18.384+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:19.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:18.904+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:19.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:18.904+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:19.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:18.904+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:19.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:05:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:05:19.345+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 283 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:05:19.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:19.295+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:19.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:19.295+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:19.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:19.295+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:19.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:19.370+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:19.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:19.370+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:19.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:19.370+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:20.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:19.876+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:20.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:19.876+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:20.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:19.876+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:20.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:20.368+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:20.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:20.368+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:20.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:20.368+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:20.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:20.328+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:20.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:20.328+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:20.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:20.328+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:21.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:20.866+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:21.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:20.866+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:21.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:20.866+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:21.633 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:21.282+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:21.633 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:21.282+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:21.633 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:21.282+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:21.633 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:21.363+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:21.633 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:21.363+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:21.633 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:21.364+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:22.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:21.871+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:22.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:21.871+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:22.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:21.871+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:22.416 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:22.305+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:22.416 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:22.305+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:22.416 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:22.305+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:22.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:22.412+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:22.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:22.412+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:22.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:22.412+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:22.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:22.412+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.14508.0:3910 3.2 3:47e04529:::20000000071.00000000:head [write 0~1120 in=1120b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:23.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:22.910+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:23.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:22.910+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:23.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:22.910+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:23.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:22.912+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:23.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:23.283+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:23.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:23.283+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:23.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:23.283+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:23.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:23.285+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:23.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:23.436+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:23.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:23.436+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:23.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:23.437+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:23.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:23.438+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:24.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:23.949+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:24.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:23.949+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:24.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:23.949+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:24.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:23.950+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:24.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:05:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:05:24.346+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 314 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:05:24.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:24.271+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:24.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:24.271+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:24.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:24.271+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:24.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:24.271+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:24.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:24.446+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:24.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:24.446+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:24.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:24.446+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:24.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:24.446+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:25.232 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:24.927+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:25.232 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:24.927+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:25.232 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:24.927+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:25.232 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:24.928+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:25.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:25.458+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:25.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:25.458+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:25.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:25.458+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:25.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:25.458+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:25.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:25.228+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:25.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:25.228+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:25.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:25.228+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:25.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:25.229+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:26.240 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:25.882+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:26.240 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:25.882+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:26.240 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:25.882+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:26.240 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:25.883+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:26.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:26.425+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:26.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:26.425+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:26.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:26.425+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:26.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:26.426+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:26.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:26.236+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:26.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:26.236+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:26.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:26.236+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:26.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:26.237+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:27.194 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:26.877+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:27.194 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:26.877+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:27.194 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:26.877+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:27.194 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:26.878+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:27.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:27.379+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:27.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:27.379+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:27.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:27.379+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:27.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:27.380+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:27.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:27.190+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:27.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:27.190+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:27.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:27.190+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:27.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:27.191+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:28.232 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:27.923+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:28.232 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:27.923+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:28.233 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:27.924+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:28.233 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:27.925+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:28.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:28.368+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:28.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:28.368+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:28.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:28.368+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:28.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:28.369+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:28.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:28.228+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:28.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:28.228+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:28.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:28.228+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:28.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:28.229+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:29.226 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:28.927+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:29.226 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:28.927+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:29.227 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:28.927+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:29.227 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:28.928+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:29.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:05:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:05:29.346+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 344 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:05:29.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:29.393+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:29.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:29.393+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:29.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:29.393+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:29.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:29.393+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:29.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:29.222+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:29.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:29.222+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:29.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:29.222+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:29.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:29.223+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:30.274 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:29.923+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:30.274 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:29.923+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:30.274 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:29.923+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:30.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:29.923+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:30.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:30.358+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:30.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:30.358+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:30.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:30.358+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:30.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:30.359+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:30.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:30.270+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:30.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:30.270+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:30.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:30.270+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:30.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:30.271+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:31.247 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:30.938+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:31.247 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:30.938+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:31.248 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:30.938+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:31.248 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:30.939+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:31.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:31.396+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:31.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:31.396+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:31.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:31.396+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:31.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:31.397+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:31.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:31.243+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:31.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:31.243+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:31.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:31.243+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:31.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:31.244+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:32.205 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:31.894+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:32.206 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:31.894+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:32.206 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:31.894+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:32.206 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:31.895+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:32.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:32.378+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:32.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:32.378+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:32.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:32.378+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:32.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:32.379+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:32.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:32.201+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:32.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:32.201+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:32.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:32.201+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:32.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:32.202+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:33.253 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:32.851+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:33.254 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:32.852+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:33.254 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:32.852+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:33.254 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:32.852+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:33.523 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:33.349+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:33.523 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:33.349+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:33.523 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:33.349+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:33.523 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:33.349+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:33.523 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:33.249+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:33.523 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:33.249+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:33.523 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:33.249+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:33.523 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:33.250+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:34.260 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:33.836+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:34.260 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:33.836+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:34.260 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:33.836+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:34.260 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:33.837+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:34.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:05:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:05:34.347+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 371 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:05:34.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:34.255+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:34.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:34.255+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:34.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:34.255+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:34.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:34.256+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:34.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:34.386+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:34.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:34.386+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:34.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:34.386+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:34.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:34.387+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:35.264 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:34.847+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:35.264 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:34.847+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:35.264 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:34.847+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:35.265 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:34.848+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:35.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:35.403+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:35.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:35.403+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:35.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:35.403+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:35.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:35.403+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:35.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:35.260+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:35.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:35.260+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:35.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:35.260+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:35.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:35.261+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:36.222 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:35.871+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:36.222 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:35.871+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:36.222 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:35.871+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:36.222 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:35.872+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:36.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:36.422+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:36.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:36.422+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:36.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:36.422+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:36.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:36.423+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:36.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:36.218+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:36.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:36.218+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:36.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:36.218+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:36.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:36.219+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:37.196 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:36.884+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:37.196 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:36.884+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:37.196 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:36.884+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:37.196 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:36.885+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:37.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:37.402+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:37.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:37.402+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:37.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:37.402+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:37.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:37.403+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:37.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:37.192+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:37.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:37.192+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:37.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:37.192+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:37.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:37.193+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:38.188 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:37.865+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:38.188 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:37.865+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:38.188 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:37.865+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:38.189 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:37.866+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:38.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:38.184+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:38.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:38.184+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:38.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:38.184+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:38.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:38.185+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:38.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:38.406+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:38.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:38.406+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:38.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:38.406+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:38.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:38.407+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:39.232 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:38.852+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:39.232 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:38.852+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:39.232 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:38.852+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:39.232 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:38.853+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:39.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:05:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:05:39.347+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 398 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:05:39.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:39.385+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:39.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:39.385+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:39.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:39.385+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:39.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:39.387+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:39.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:39.228+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:39.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:39.228+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:39.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:39.228+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:39.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:39.229+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:40.233 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:39.808+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:40.233 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:39.808+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:40.233 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:39.808+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:40.233 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:39.809+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:40.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:40.366+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:40.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:40.366+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:40.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:40.366+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:40.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:40.367+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:40.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:40.229+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:40.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:40.229+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:40.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:40.229+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:40.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:40.230+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:41.209 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:40.790+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:41.209 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:40.790+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:41.209 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:40.790+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:41.209 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:40.791+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:41.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:41.382+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:41.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:41.383+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:41.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:41.383+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:41.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:41.383+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:41.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:41.205+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:41.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:41.205+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:41.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:41.205+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:41.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:41.206+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:42.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:41.745+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:42.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:41.745+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:42.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:41.745+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:42.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:41.746+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:42.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:42.355+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:42.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:42.355+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:42.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:42.355+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:42.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:42.356+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:42.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:42.194+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:42.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:42.194+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:42.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:42.194+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:42.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:42.194+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:43.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:42.711+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:43.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:42.711+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:43.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:42.711+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:43.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:42.712+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:43.524 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:43.327+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:43.524 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:43.327+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:43.524 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:43.327+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:43.524 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:43.328+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:43.524 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:43.175+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:43.524 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:43.175+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:43.524 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:43.175+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:43.524 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:43.175+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:43.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:43.671+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:43.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:43.671+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:43.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:43.671+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:43.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:43.672+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:44.475 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:05:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:05:44.347+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 426 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:05:44.475 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:44.339+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:44.475 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:44.339+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:44.475 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:44.339+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:44.475 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:44.340+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:44.476 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:44.164+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:44.476 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:44.164+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:44.476 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:44.164+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:44.476 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:44.165+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:44.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:44.650+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:44.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:44.650+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:44.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:44.650+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:44.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:44.651+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:45.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:45.183+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:45.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:45.183+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:45.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:45.183+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:45.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:45.184+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:45.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:45.336+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:45.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:45.336+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:45.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:45.336+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:45.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:45.337+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:46.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:45.605+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:46.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:45.605+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:46.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:45.605+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:46.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:45.606+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:46.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:46.320+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:46.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:46.320+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:46.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:46.320+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:46.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:46.321+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:46.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:46.144+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:46.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:46.144+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:46.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:46.144+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:46.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:46.145+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:47.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:46.597+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:47.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:46.597+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:47.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:46.597+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:47.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:46.598+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:47.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:47.307+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:47.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:47.307+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:47.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:47.307+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:47.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:47.308+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:47.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:47.191+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:47.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:47.191+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:47.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:47.191+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:47.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:47.191+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:48.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:47.579+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:48.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:47.579+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:48.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:47.579+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:48.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:47.580+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:48.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:48.285+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:48.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:48.285+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:48.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:48.285+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:48.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:48.286+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:48.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:48.192+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:48.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:48.192+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:48.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:48.192+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:48.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:48.193+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:49.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:48.614+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:49.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:48.614+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:49.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:48.614+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:49.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:48.615+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:49.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:05:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:05:49.348+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 454 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:05:49.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:49.234+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:49.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:49.234+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:49.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:49.234+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:49.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:49.235+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:49.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:49.312+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:49.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:49.312+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:49.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:49.312+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:49.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:49.313+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:50.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:49.569+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:50.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:49.569+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:50.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:49.569+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:50.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:49.569+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:50.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:50.361+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:50.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:50.361+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:50.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:50.361+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:50.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:50.362+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:50.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:50.258+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:50.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:50.258+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:50.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:50.258+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:50.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:50.259+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:51.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:50.593+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:51.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:50.593+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:51.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:50.593+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:51.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:50.594+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:51.594 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:51.389+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:51.594 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:51.389+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:51.594 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:51.389+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:51.594 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:51.389+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:51.594 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:51.293+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:51.595 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:51.293+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:51.595 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:51.293+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:51.595 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:51.294+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:52.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:51.590+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:52.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:51.590+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:52.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:51.590+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:52.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:51.591+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:52.549 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:52.405+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:52.550 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:52.405+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:52.550 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:52.405+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:52.550 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:52.406+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:52.550 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:52.289+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:52.550 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:52.289+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:52.550 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:52.289+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:52.550 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:52.290+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:52.806 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:52.545+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:52.806 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:52.545+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:52.806 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:52.545+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:52.806 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:52.545+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:53.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:53.431+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:53.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:53.431+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:53.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:53.431+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:53.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:53.431+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:53.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:53.270+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:53.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:53.270+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:53.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:53.270+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:53.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:53.271+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:53.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:53.498+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:53.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:53.498+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:53.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:53.498+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:53.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:53.499+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:54.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:05:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:05:54.348+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 481 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:05:54.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:54.460+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:54.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:54.460+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:54.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:54.460+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:54.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:54.461+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:54.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:54.382+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:54.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:54.382+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:54.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:54.382+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:54.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:54.383+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:54.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:54.289+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:54.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:54.289+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:54.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:54.289+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:54.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:54.290+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:55.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:55.353+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:55.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:55.353+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:55.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:55.353+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:55.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:55.353+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:55.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:55.441+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:55.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:55.441+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:55.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:55.441+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:55.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:55.441+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:55.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:55.337+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:55.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:55.337+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:55.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:55.337+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:55.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:55.338+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:56.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:56.464+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:56.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:56.464+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:56.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:56.464+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:56.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:56.465+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:56.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:56.395+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:56.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:56.395+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:56.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:56.395+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:56.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:56.395+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:56.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:56.304+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:56.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:56.304+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:56.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:56.304+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:56.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:56.305+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:57.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:57.432+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:57.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:57.432+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:57.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:57.432+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:57.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:57.433+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:57.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:57.465+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:57.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:57.465+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:57.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:57.465+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:57.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:57.466+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:57.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:57.332+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:57.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:57.332+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:57.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:57.332+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:57.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:57.333+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:58.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:58.480+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:58.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:58.480+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:58.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:58.480+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:58.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:58.481+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:58.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:58.360+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:58.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:58.360+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:58.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:58.360+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:58.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:58.361+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:58.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:58.440+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:58.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:58.440+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:58.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:58.440+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:58.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:58.441+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:59.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:05:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:05:59.349+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 508 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:05:59.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:59.512+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:59.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:59.512+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:59.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:59.512+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:05:59.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:05:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:05:59.513+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:59.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:59.332+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:59.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:59.332+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:59.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:59.332+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:05:59.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:05:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:05:59.333+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:05:59.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:59.471+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:59.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:59.471+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:59.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:59.471+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:05:59.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:05:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:05:59.472+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:00.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:00.488+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:00.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:00.489+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:00.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:00.489+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:00.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:00.489+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:00.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:00.472+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:00.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:00.472+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:00.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:00.472+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:00.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:00.473+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:00.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:00.306+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:00.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:00.306+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:00.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:00.306+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:00.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:00.307+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:01.636 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:01.428+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:01.636 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:01.428+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:01.636 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:01.428+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:01.636 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:01.429+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:01.636 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:01.466+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:01.637 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:01.466+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:01.637 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:01.466+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:01.637 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:01.467+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:01.637 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:01.322+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:01.637 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:01.322+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:01.637 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:01.322+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:01.637 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:01.323+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:02.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:02.426+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:02.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:02.426+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:02.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:02.426+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:02.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:02.427+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:02.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:02.406+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:02.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:02.406+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:02.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:02.406+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:02.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:02.407+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:02.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:02.298+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:02.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:02.298+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:02.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:02.298+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:02.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:02.299+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:03.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:03.438+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:03.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:03.438+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:03.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:03.438+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:03.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:03.439+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:03.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:03.415+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:03.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:03.415+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:03.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:03.415+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:03.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:03.416+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:03.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:03.330+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:03.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:03.330+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:03.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:03.330+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:03.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:03.331+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:04.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:04.391+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:04.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:04.391+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:04.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:04.391+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:04.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:04.392+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:04.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:04.434+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:04.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:04.434+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:04.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:04.434+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:04.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:04.435+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:04.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:04.317+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:04.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:04.317+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:04.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:04.317+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:04.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:04.318+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:04.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:06:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:06:04.349+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 536 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:06:05.412 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:05.310+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:05.412 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:05.310+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:05.412 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:05.310+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:05.412 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:05.311+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:05.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:05.415+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:05.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:05.415+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:05.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:05.415+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:05.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:05.416+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:05.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:05.408+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:05.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:05.408+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:05.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:05.408+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:05.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:05.409+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:06.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:06.454+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:06.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:06.454+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:06.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:06.454+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:06.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:06.455+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:06.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:06.293+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:06.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:06.293+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:06.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:06.293+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:06.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:06.293+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:06.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:06.393+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:06.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:06.393+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:06.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:06.393+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:06.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:06.395+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:07.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:07.426+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:07.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:07.426+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:07.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:07.426+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:07.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:07.427+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:07.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:07.485+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:07.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:07.485+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:07.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:07.485+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:07.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:07.486+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:07.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:07.279+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:07.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:07.279+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:07.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:07.279+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:07.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:07.280+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:08.453 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:08.299+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:08.453 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:08.299+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:08.453 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:08.299+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:08.453 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:08.299+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:08.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:08.460+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:08.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:08.460+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:08.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:08.460+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:08.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:08.461+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:08.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:08.449+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:08.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:08.449+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:08.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:08.449+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:08.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:08.450+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:09.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:06:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:06:09.350+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 564 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:06:09.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:09.449+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:09.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:09.449+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:09.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:09.449+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:09.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:09.449+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:09.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:09.266+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:09.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:09.266+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:09.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:09.266+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:09.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:09.267+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:09.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:09.445+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:09.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:09.445+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:09.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:09.445+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:09.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:09.446+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:10.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:10.412+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:10.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:10.412+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:10.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:10.412+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:10.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:10.413+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:10.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:10.441+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:10.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:10.441+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:10.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:10.441+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:10.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:10.442+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:10.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:10.265+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:10.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:10.265+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:10.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:10.265+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:10.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:10.266+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:11.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:11.452+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:11.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:11.452+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:11.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:11.452+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:11.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:11.453+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:11.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:11.464+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:11.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:11.464+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:11.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:11.464+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:11.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:11.465+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:11.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:11.258+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:11.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:11.258+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:11.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:11.258+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:11.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:11.259+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:12.489 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:12.448+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:12.489 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:12.448+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:12.489 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:12.448+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:12.489 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:12.448+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:12.490 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:12.230+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:12.490 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:12.230+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:12.490 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:12.230+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:12.490 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:12.231+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:12.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:12.485+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:12.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:12.485+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:12.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:12.485+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:12.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:12.486+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:13.509 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:13.412+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:13.509 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:13.412+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:13.509 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:13.412+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:13.509 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:13.412+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:13.509 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:13.224+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:13.510 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:13.224+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:13.510 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:13.224+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:13.510 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:13.225+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:13.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:13.505+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:13.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:13.505+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:13.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:13.505+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:13.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:13.506+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:14.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:06:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:06:14.350+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 594 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:06:14.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:14.263+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:14.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:14.263+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:14.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:14.263+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:14.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:14.264+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:14.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:14.442+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:14.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:14.442+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:14.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:14.442+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:14.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:14.443+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:15.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:14.550+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:15.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:14.550+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:15.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:14.550+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:15.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:14.551+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:15.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:15.475+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:15.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:15.475+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:15.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:15.475+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:15.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:15.475+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:15.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:15.529+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:15.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:15.529+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:15.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:15.529+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:15.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:15.530+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:15.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:15.308+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:15.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:15.308+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:15.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:15.308+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:15.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:15.308+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:16.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:16.459+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:16.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:16.459+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:16.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:16.459+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:16.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:16.460+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:16.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:16.261+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:16.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:16.261+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:16.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:16.261+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:16.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:16.262+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:16.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:16.487+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:16.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:16.487+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:16.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:16.487+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:16.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:16.487+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:17.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:17.269+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:17.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:17.269+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:17.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:17.269+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:17.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:17.269+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:17.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:17.446+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:17.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:17.446+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:17.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:17.446+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:17.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:17.447+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:18.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:17.534+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:18.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:17.534+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:18.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:17.534+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:18.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:17.534+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:18.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:18.471+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:18.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:18.471+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:18.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:18.471+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:18.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:18.472+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:18.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:18.263+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:18.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:18.263+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:18.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:18.263+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:18.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:18.264+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:19.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:18.581+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:19.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:18.581+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:19.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:18.581+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:19.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:18.582+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:19.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:06:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:06:19.351+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 627 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:06:19.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:19.463+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:19.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:19.463+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:19.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:19.463+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:19.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:19.464+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:19.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:19.251+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:19.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:19.251+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:19.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:19.251+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:19.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:19.252+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:20.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:19.547+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:20.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:19.547+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:20.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:19.547+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:20.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:19.548+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:20.487 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:20.207+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:20.487 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:20.207+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:20.487 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:20.207+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:20.487 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:20.208+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:20.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:20.482+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:20.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:20.482+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:20.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:20.482+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:20.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:20.483+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:20.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:20.556+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:20.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:20.556+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:20.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:20.556+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:20.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:20.557+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:21.514 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:21.212+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:21.514 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:21.212+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:21.514 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:21.212+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:21.514 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:21.213+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:21.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:21.574+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:21.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:21.574+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:21.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:21.574+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:21.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:21.575+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:21.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:21.510+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:21.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:21.510+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:21.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:21.510+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:21.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:21.510+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:22.488 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:22.214+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:22.488 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:22.214+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:22.488 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:22.214+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:22.488 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:22.214+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:22.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:22.620+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:22.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:22.620+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:22.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:22.620+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:22.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:22.621+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:22.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:22.485+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:22.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:22.485+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:22.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:22.485+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:22.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:22.485+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:23.518 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:23.250+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:23.518 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:23.250+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:23.519 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:23.250+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:23.519 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:23.250+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:23.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:23.634+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:23.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:23.634+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:23.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:23.634+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:23.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:23.635+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:23.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:23.515+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:23.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:23.515+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:23.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:23.515+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:23.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:23.516+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:24.637 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:06:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:06:24.351+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 655 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:06:24.637 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:24.482+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:24.637 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:24.482+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:24.637 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:24.482+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:24.637 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:24.484+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:24.638 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:24.283+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:24.638 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:24.283+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:24.638 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:24.283+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:24.638 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:24.284+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:25.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:24.633+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:25.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:24.633+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:25.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:24.633+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:25.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:24.634+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:25.622 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:25.477+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:25.623 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:25.477+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:25.623 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:25.477+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:25.623 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:25.477+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:25.623 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:25.326+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:25.623 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:25.326+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:25.623 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:25.326+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:25.623 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:25.327+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:26.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:25.618+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:26.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:25.618+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:26.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:25.618+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:26.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:25.619+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:26.617 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:26.527+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:26.617 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:26.527+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:26.617 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:26.527+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:26.617 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:26.527+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:26.617 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:26.353+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:26.617 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:26.353+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:26.617 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:26.353+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:26.617 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:26.353+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:27.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:26.613+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:27.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:26.613+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:27.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:26.613+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:27.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:26.613+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:27.626 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:27.321+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:27.626 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:27.321+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:27.626 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:27.321+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:27.626 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:27.321+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:27.626 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:27.545+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:27.626 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:27.545+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:27.626 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:27.545+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:27.626 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:27.547+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:28.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:27.621+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:28.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:27.621+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:28.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:27.621+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:28.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:27.623+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:28.641 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:28.542+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:28.641 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:28.542+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:28.641 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:28.542+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:28.641 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:28.543+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:28.641 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:28.360+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:28.641 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:28.360+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:28.641 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:28.360+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:28.641 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:28.361+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:29.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:28.637+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:29.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:28.637+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:29.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:28.637+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:29.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:28.637+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:29.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:06:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:06:29.352+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 683 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:06:29.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:29.600+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:29.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:29.600+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:29.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:29.600+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:29.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:29.601+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:29.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:29.510+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:29.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:29.510+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:29.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:29.510+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:29.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:29.511+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:29.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:29.359+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:29.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:29.359+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:29.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:29.359+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:29.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:29.360+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:30.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:30.554+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:30.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:30.554+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:30.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:30.554+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:30.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:30.555+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:30.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:30.520+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:30.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:30.520+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:30.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:30.520+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:30.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:30.521+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:30.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:30.364+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:30.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:30.364+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:30.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:30.364+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:30.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:30.365+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:31.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:31.407+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:31.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:31.407+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:31.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:31.407+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:31.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:31.408+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:31.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:31.594+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:31.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:31.594+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:31.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:31.594+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:31.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:31.595+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:31.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:31.474+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:31.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:31.474+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:31.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:31.474+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:31.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:31.475+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:32.742 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:32.524+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:32.743 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:32.524+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:32.743 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:32.524+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:32.743 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:32.525+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:32.743 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:32.410+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:32.743 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:32.410+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:32.743 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:32.410+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:32.743 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:32.411+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:32.744 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:32.569+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:32.744 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:32.569+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:32.744 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:32.569+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:32.744 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:32.569+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:33.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:33.605+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:33.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:33.605+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:33.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:33.605+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:33.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:33.605+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:33.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:33.559+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:33.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:33.559+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:33.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:33.559+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:33.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:33.561+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:33.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:33.442+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:33.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:33.442+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:33.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:33.442+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:33.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:33.443+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:34.647 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:06:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:06:34.352+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 710 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:06:34.648 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:34.584+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:34.648 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:34.584+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:34.648 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:34.584+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:34.648 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:34.585+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:34.649 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:34.485+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:34.649 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:34.485+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:34.649 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:34.485+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:34.649 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:34.486+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:35.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:34.643+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:35.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:34.643+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:35.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:34.643+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:35.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:34.644+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:35.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:35.601+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:35.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:35.601+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:35.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:35.601+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:35.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:35.602+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:35.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:35.553+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:35.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:35.553+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:35.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:35.553+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:35.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:35.554+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:35.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:35.458+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:35.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:35.458+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:35.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:35.458+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:35.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:35.459+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:36.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:36.618+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:36.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:36.618+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:36.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:36.618+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:36.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:36.619+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:36.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:36.602+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:36.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:36.602+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:36.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:36.602+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:36.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:36.603+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:36.782 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:36.478+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:36.782 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:36.478+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:36.782 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:36.478+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:36.782 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:36.479+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:37.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:37.596+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:37.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:37.596+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:37.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:37.596+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:37.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:37.597+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:37.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:37.582+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:37.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:37.582+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:37.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:37.582+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:37.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:37.582+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:37.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:37.465+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:37.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:37.465+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:37.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:37.465+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:37.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:37.465+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:38.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:38.624+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:38.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:38.624+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:38.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:38.624+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:38.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:38.625+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:38.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:38.572+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:38.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:38.572+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:38.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:38.572+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:38.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:38.573+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:38.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:38.477+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:38.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:38.477+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:38.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:38.477+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:38.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:38.478+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:39.626 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:06:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:06:39.353+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 734 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:06:39.626 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:39.491+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:39.626 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:39.491+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:39.626 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:39.491+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:39.626 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:39.492+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:39.627 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:39.581+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:39.627 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:39.581+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:39.627 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:39.581+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:39.627 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:39.582+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:40.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:39.622+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:40.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:39.622+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:40.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:39.622+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:40.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:39.623+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:41.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:40.533+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:41.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:40.533+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:41.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:40.533+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:41.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:40.535+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:41.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:40.599+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:41.031 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:40.599+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:41.031 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:40.599+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:41.031 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:40.599+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:41.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:40.593+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:41.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:40.593+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:41.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:40.593+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:41.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:40.594+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:41.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:41.577+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:41.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:41.577+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:41.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:41.577+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:41.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:41.578+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:41.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:41.521+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:41.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:41.521+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:41.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:41.521+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:41.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:41.522+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:41.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:41.549+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:41.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:41.549+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:41.782 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:41.549+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:41.782 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:41.550+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:43.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:42.597+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:43.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:42.597+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:43.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:42.597+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:43.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:42.598+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:43.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:42.567+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:43.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:42.567+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:43.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:42.567+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:43.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:42.568+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:43.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:42.539+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:43.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:42.539+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:43.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:42.539+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:43.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:42.540+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:43.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:43.581+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:43.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:43.581+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:43.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:43.581+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:43.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:43.582+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:43.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:43.578+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:43.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:43.578+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:43.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:43.578+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:43.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:43.579+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:43.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:43.532+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:43.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:43.532+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:43.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:43.532+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:43.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:43.533+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:44.631 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:06:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:06:44.353+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 765 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:06:44.632 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:44.554+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:44.632 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:44.554+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:44.632 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:44.554+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:44.632 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:44.555+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:44.632 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:44.554+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:44.632 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:44.554+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:44.632 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:44.554+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:44.632 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:44.555+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:45.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:44.627+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:45.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:44.627+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:45.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:44.627+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:45.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:44.628+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:45.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:45.597+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:45.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:45.597+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:45.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:45.597+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:45.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:45.598+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:45.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:45.525+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:45.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:45.525+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:45.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:45.525+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:45.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:45.526+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:45.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:45.553+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:45.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:45.553+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:45.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:45.553+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:45.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:45.554+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:47.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:46.593+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:47.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:46.593+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:47.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:46.593+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:47.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:46.594+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:47.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:46.571+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:47.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:46.571+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:47.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:46.571+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:47.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:46.572+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:47.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:46.563+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:47.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:46.563+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:47.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:46.563+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:47.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:46.564+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:48.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:47.596+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:48.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:47.596+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:48.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:47.596+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:48.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:47.597+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:48.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:47.602+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:48.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:47.602+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:48.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:47.602+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:48.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:47.603+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:48.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:47.575+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:48.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:47.575+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:48.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:47.575+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:48.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:47.576+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:49.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:48.571+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:49.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:48.571+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:49.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:48.571+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:49.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:48.572+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:49.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:48.570+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:49.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:48.570+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:49.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:48.570+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:49.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:48.571+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:49.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:48.582+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:49.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:48.582+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:49.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:48.582+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:49.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:48.583+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:49.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:06:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:06:49.354+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 792 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:06:49.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:49.564+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:49.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:49.564+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:49.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:49.564+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:49.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:49.565+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:49.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:49.560+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:49.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:49.560+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:49.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:49.560+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:49.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:49.561+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:49.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:49.521+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:49.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:49.521+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:49.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:49.521+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:49.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:49.522+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:51.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:50.561+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:51.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:50.561+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:51.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:50.561+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:51.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:50.562+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:51.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:50.534+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:51.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:50.534+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:51.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:50.534+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:51.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:50.535+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:51.031 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:50.542+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:51.031 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:50.542+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:51.031 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:50.542+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:51.032 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:50.543+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:52.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:51.575+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:52.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:51.575+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:52.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:51.575+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:52.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:51.576+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:52.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:51.535+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:52.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:51.535+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:52.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:51.535+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:52.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:51.535+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:52.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:51.567+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:52.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:51.567+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:52.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:51.567+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:52.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:51.568+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:52.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:52.568+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:52.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:52.568+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:52.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:52.568+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:52.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:52.569+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:52.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:52.521+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:52.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:52.521+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:52.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:52.521+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:52.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:52.522+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:52.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:52.543+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:52.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:52.543+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:52.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:52.543+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:52.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:52.544+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:53.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:53.550+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:53.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:53.550+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:53.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:53.550+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:53.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:53.550+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:53.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:53.606+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:53.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:53.606+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:53.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:53.606+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:53.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:53.607+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:53.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:53.540+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:53.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:53.541+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:53.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:53.541+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:53.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:53.541+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:54.505 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:06:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:06:54.354+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 820 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:06:54.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:54.623+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:54.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:54.623+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:54.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:54.623+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:54.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:54.624+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:54.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:54.555+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:54.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:54.555+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:54.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:54.555+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:54.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:54.556+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:54.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:54.501+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:54.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:54.501+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:54.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:54.501+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:54.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:54.502+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:56.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:55.616+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:56.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:55.616+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:56.031 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:55.616+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:56.031 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:55.617+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:56.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:55.589+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:56.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:55.589+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:56.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:55.589+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:56.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:55.590+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:56.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:55.525+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:56.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:55.526+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:56.032 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:55.526+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:56.032 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:55.527+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:57.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:56.589+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:57.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:56.589+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:57.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:56.589+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:57.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:56.589+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:57.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:56.646+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:57.031 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:56.646+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:57.031 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:56.646+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:57.031 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:56.647+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:57.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:56.554+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:57.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:56.554+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:57.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:56.554+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:57.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:56.555+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:58.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:57.618+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:58.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:57.618+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:58.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:57.618+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:58.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:57.619+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:58.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:57.610+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:58.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:57.610+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:58.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:57.610+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:58.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:57.611+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:58.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:57.551+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:58.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:57.551+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:58.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:57.551+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:58.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:57.551+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:59.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:58.574+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:59.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:58.574+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:59.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:58.574+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:59.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:58.575+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:59.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:58.584+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:59.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:58.584+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:59.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:58.584+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:06:59.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:58.585+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:59.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:58.531+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:59.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:58.531+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:59.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:58.531+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:59.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:58.532+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:59.627 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:59.558+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:59.627 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:59.558+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:59.627 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:59.558+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:06:59.627 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:06:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:06:59.558+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:59.628 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:59.575+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:59.628 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:59.575+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:59.628 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:59.575+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:06:59.628 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:06:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:06:59.576+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:06:59.628 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:06:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:06:59.355+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 846 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:07:00.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:59.623+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:00.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:59.623+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:00.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:59.623+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:00.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:06:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:06:59.624+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:01.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:00.546+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:01.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:00.546+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:01.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:00.546+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:01.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:00.547+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:01.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:00.596+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:01.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:00.596+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:01.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:00.596+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:01.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:00.596+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:01.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:00.564+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:01.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:00.564+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:01.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:00.564+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:01.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:00.564+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:01.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:01.572+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:01.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:01.572+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:01.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:01.572+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:01.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:01.573+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:01.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:01.565+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:01.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:01.565+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:01.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:01.565+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:01.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:01.566+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:01.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:01.515+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:01.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:01.515+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:01.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:01.515+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:01.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:01.516+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:02.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:02.532+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:02.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:02.532+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:02.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:02.532+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:02.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:02.533+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:02.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:02.564+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:02.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:02.564+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:02.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:02.564+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:02.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:02.565+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:02.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:02.474+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:02.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:02.474+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:02.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:02.474+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:02.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:02.475+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:03.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:03.535+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:03.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:03.535+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:03.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:03.535+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:03.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:03.536+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:03.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:03.539+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:03.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:03.539+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:03.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:03.539+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:03.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:03.540+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:03.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:03.481+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:03.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:03.481+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:03.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:03.481+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:03.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:03.481+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:04.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:07:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:07:04.355+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 875 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:07:04.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:04.502+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:04.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:04.502+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:04.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:04.502+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:04.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:04.503+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:04.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:04.562+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:04.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:04.562+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:04.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:04.562+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:04.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:04.562+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:04.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:04.446+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:04.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:04.446+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:04.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:04.446+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:04.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:04.447+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:05.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:05.485+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:05.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:05.485+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:05.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:05.485+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:05.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:05.486+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:05.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:05.560+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:05.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:05.560+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:05.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:05.560+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:05.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:05.561+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:05.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:05.446+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:05.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:05.446+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:05.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:05.446+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:05.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:05.447+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:06.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:06.497+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:06.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:06.497+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:06.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:06.497+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:06.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:06.498+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:06.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:06.474+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:06.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:06.474+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:06.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:06.474+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:06.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:06.475+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:06.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:06.534+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:06.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:06.534+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:06.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:06.534+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:06.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:06.535+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:07.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:07.481+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:07.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:07.481+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:07.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:07.481+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:07.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:07.482+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:07.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:07.561+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:07.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:07.561+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:07.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:07.561+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:07.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:07.561+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:07.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:07.486+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:07.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:07.486+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:07.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:07.486+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:07.782 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:07.487+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:08.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:08.453+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:08.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:08.453+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:08.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:08.453+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:08.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:08.454+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:08.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:08.552+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:08.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:08.552+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:08.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:08.552+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:08.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:08.553+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:08.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:08.451+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:08.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:08.451+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:08.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:08.451+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:08.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:08.453+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:09.681 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:07:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:07:09.356+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 902 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:07:09.681 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:09.468+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:09.681 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:09.468+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:09.681 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:09.468+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:09.681 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:09.469+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:09.682 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:09.433+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:09.682 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:09.433+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:09.682 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:09.433+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:09.682 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:09.434+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:09.682 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:09.593+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:09.682 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:09.593+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:09.682 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:09.594+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:09.682 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:09.595+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:10.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:10.435+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:10.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:10.435+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:10.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:10.435+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:10.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:10.435+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:10.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:10.570+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:10.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:10.570+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:10.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:10.570+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:10.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:10.571+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:10.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:10.487+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:10.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:10.487+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:10.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:10.487+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:10.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:10.487+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:11.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:11.388+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:11.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:11.388+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:11.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:11.388+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:11.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:11.389+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:11.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:11.582+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:11.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:11.582+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:11.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:11.582+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:11.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:11.583+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:11.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:11.457+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:11.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:11.457+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:11.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:11.457+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:11.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:11.457+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:12.608 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:12.345+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:12.608 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:12.345+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:12.608 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:12.345+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:12.608 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:12.346+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:12.608 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:12.443+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:12.609 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:12.443+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:12.609 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:12.443+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:12.609 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:12.443+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:13.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:12.603+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:13.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:12.603+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:13.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:12.604+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:13.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:12.605+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:13.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:13.587+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:13.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:13.587+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:13.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:13.587+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:13.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:13.588+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:13.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:13.482+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:13.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:13.482+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:13.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:13.482+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:13.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:13.483+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:13.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:13.355+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:13.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:13.355+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:13.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:13.355+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:13.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:13.356+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:14.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:07:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:07:14.357+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 930 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:07:14.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:14.564+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:14.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:14.564+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:14.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:14.564+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:14.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:14.565+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:14.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:14.438+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:14.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:14.438+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:14.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:14.438+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:14.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:14.439+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:14.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:14.361+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:14.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:14.361+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:14.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:14.361+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:14.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:14.363+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:15.751 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:15.383+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:15.751 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:15.383+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:15.751 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:15.383+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:15.751 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:15.384+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:15.752 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:15.609+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:15.752 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:15.609+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:15.752 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:15.609+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:15.752 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:15.609+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:15.752 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:15.414+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:15.752 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:15.414+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:15.752 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:15.414+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:15.752 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:15.415+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:16.646 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:16.377+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:16.646 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:16.377+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:16.646 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:16.377+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:16.646 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:16.378+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:16.646 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:16.430+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:16.646 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:16.430+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:16.646 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:16.430+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:16.646 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:16.430+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:17.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:16.641+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:17.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:16.641+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:17.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:16.641+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:17.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:16.642+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:17.665 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:17.399+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:17.665 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:17.399+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:17.665 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:17.399+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:17.666 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:17.400+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:17.666 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:17.387+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:17.666 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:17.387+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:17.666 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:17.387+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:17.666 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:17.388+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:17.666 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:17.661+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:18.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:17.661+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:18.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:17.661+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:18.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:17.662+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:18.627 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:18.350+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:18.627 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:18.350+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:18.627 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:18.350+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:18.627 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:18.350+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:18.627 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:18.408+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:18.627 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:18.408+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:18.628 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:18.408+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:18.628 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:18.408+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:19.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:18.622+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:19.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:18.622+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:19.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:18.623+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:19.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:18.624+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:19.614 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:19.367+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:19.614 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:19.367+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:19.614 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:19.367+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:19.614 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:19.368+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:19.615 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:19.388+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:19.615 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:19.388+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:19.615 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:19.388+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:19.615 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:19.389+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:19.615 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:07:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:07:19.358+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 956 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:07:20.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:19.611+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:20.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:19.611+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:20.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:19.611+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:20.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:19.611+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:20.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:20.387+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:20.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:20.387+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:20.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:20.387+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:20.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:20.388+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:20.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:20.631+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:20.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:20.631+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:20.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:20.631+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:20.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:20.631+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:20.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:20.410+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:20.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:20.410+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:20.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:20.410+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:20.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:20.410+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:21.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:21.406+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:21.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:21.406+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:21.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:21.406+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:21.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:21.407+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:21.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:21.597+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:21.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:21.597+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:21.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:21.597+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:21.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:21.598+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:21.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:21.425+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:21.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:21.425+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:21.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:21.425+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:21.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:21.426+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:22.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:22.565+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:22.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:22.565+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:22.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:22.565+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:22.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:22.565+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:22.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:22.384+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:22.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:22.384+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:22.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:22.384+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:22.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:22.385+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:22.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:22.425+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:22.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:22.425+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:22.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:22.425+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:22.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:22.426+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:23.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:23.570+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:23.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:23.570+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:23.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:23.570+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:23.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:23.571+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:23.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:23.430+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:23.782 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:23.430+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:23.782 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:23.430+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:23.782 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:23.431+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:23.782 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:23.464+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:23.782 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:23.464+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:23.782 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:23.464+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:23.782 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:23.465+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:24.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:07:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:07:24.358+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 992 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:07:24.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:24.576+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:24.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:24.576+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:24.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:24.576+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:24.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:24.577+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:24.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:24.442+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:24.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:24.442+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:24.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:24.442+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:24.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:24.443+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:24.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:24.442+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:24.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:24.442+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:24.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:24.442+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:24.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:24.443+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:25.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:25.545+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:25.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:25.545+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:25.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:25.545+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:25.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:25.546+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:25.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:25.473+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:25.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:25.473+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:25.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:25.473+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:25.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:25.474+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:25.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:25.489+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:25.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:25.489+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:25.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:25.489+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:25.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:25.490+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:26.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:26.500+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:26.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:26.500+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:26.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:26.500+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:26.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:26.500+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:26.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:26.450+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:26.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:26.450+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:26.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:26.450+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:26.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:26.451+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:26.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:26.493+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:26.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:26.493+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:26.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:26.493+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:26.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:26.494+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:27.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:27.496+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:27.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:27.496+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:27.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:27.496+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:27.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:27.497+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:27.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:27.443+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:27.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:27.443+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:27.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:27.443+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:27.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:27.444+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:27.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:27.457+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:27.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:27.457+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:27.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:27.457+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:27.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:27.458+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:28.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:28.419+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:28.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:28.419+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:28.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:28.419+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:28.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:28.420+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:28.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:28.472+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:28.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:28.472+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:28.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:28.472+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:28.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:28.473+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:28.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:28.530+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:28.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:28.530+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:28.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:28.530+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:28.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:28.531+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:29.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:07:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:07:29.359+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 1035 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:07:29.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:29.399+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:29.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:29.399+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:29.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:29.399+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:29.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:29.400+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:29.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:29.476+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:29.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:29.476+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:29.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:29.476+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:29.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:29.477+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:29.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:29.578+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:29.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:29.578+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:29.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:29.578+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:29.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:29.579+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:30.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:30.448+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:30.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:30.448+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:30.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:30.448+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:30.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:30.449+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:30.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:30.443+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:30.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:30.443+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:30.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:30.443+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:30.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:30.444+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:30.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:30.619+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:30.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:30.619+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:30.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:30.619+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:30.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:30.620+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:31.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:31.476+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:31.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:31.476+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:31.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:31.476+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:31.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:31.477+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:31.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:31.448+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:31.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:31.448+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:31.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:31.448+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:31.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:31.450+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:31.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:31.589+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:31.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:31.589+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:31.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:31.589+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:31.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:31.590+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:32.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:07:32.259+0000 7fcc30afa700 0 monclient(hunting): authenticate timed out after 300 2024-03-11T23:07:32.263 INFO:teuthology.orchestra.run.smithi153.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-03-11T23:07:32.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:32.497+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:32.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:32.497+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:32.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:32.497+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:32.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:32.498+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:32.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:32.442+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:32.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:32.442+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:32.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:32.442+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:32.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:32.443+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:32.986 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:32.547+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:32.986 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:32.547+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:32.986 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:32.547+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:32.986 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:32.548+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:33.011 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-03-11T23:07:33.289 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T23:07:33.671 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:33.533+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:33.672 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:33.533+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:33.672 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:33.533+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:33.672 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:33.534+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:33.673 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:33.451+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:33.673 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:33.451+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:33.673 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:33.451+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:33.673 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:33.452+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:33.674 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:33.453+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:33.674 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:33.453+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:33.674 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:33.453+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:33.674 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:33.454+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:34.669 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:07:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:07:34.359+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 1079 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:07:34.670 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:34.489+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:34.670 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:34.489+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:34.670 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:34.489+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:34.670 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:34.490+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:34.670 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:34.568+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:34.670 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:34.568+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:34.670 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:34.568+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:34.670 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:34.569+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:34.671 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:34.453+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:34.671 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:34.453+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:34.671 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:34.453+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:34.671 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:34.454+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:35.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:35.612+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:35.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:35.612+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:35.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:35.612+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:35.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:35.613+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:35.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:35.445+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:35.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:35.445+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:35.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:35.445+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:35.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:35.446+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:35.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:35.452+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:35.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:35.452+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:35.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:35.452+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:35.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:35.453+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:36.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:36.623+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:36.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:36.623+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:36.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:36.623+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:36.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:36.624+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:36.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:36.476+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:36.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:36.476+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:36.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:36.476+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:36.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:36.477+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:36.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:36.464+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:36.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:36.464+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:36.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:36.464+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:36.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:36.465+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:37.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:37.621+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:37.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:37.621+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:37.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:37.621+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:37.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:37.622+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:37.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:37.416+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:37.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:37.416+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:37.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:37.416+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:37.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:37.416+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:37.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:37.501+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:37.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:37.501+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:37.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:37.501+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:37.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:37.502+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:38.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:38.435+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:38.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:38.435+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:38.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:38.435+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:38.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:38.435+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:38.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:38.645+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:38.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:38.645+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:38.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:38.645+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:38.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:38.646+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:38.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:38.501+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:38.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:38.501+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:38.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:38.501+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:38.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:38.502+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:39.639 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:07:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:07:39.360+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 1121 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:07:39.639 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:39.473+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:39.639 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:39.473+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:39.639 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:39.473+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:39.639 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:39.474+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:39.640 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:39.549+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:39.640 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:39.549+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:39.640 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:39.549+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:39.640 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:39.550+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:40.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:39.635+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:40.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:39.635+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:40.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:39.635+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:40.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:39.635+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:40.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:40.608+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:40.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:40.608+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:40.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:40.608+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:40.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:40.609+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:40.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:40.533+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:40.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:40.533+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:40.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:40.533+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:40.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:40.534+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:40.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:40.483+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:40.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:40.483+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:40.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:40.483+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:40.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:40.483+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:41.508 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:41.502+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:41.508 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:41.502+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:41.509 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:41.502+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:41.509 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:41.503+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:41.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:41.505+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:41.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:41.505+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:41.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:41.505+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:41.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:41.506+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:41.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:41.622+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:41.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:41.622+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:41.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:41.622+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:41.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:41.623+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:42.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:42.471+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:42.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:42.471+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:42.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:42.471+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:42.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:42.472+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:42.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:42.658+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:42.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:42.658+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:42.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:42.658+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:42.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:42.658+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:42.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:42.530+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:42.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:42.530+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:42.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:42.530+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:42.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:42.531+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:43.697 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:43.501+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:43.697 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:43.501+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:43.697 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:43.501+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:43.697 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:43.502+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:43.698 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:43.432+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:43.698 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:43.432+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:43.698 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:43.432+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:43.698 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:43.432+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:44.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:43.693+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:44.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:43.693+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:44.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:43.693+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:44.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:43.693+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:44.681 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:07:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:07:44.360+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 1177 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:07:44.681 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:44.412+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:44.681 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:44.412+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:44.682 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:44.412+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:44.682 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:44.412+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:44.682 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:44.544+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:44.682 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:44.544+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:44.682 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:44.544+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:44.682 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:44.545+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:45.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:44.677+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:45.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:44.677+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:45.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:44.677+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:45.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:44.678+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:45.693 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:45.553+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:45.693 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:45.553+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:45.693 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:45.553+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:45.693 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:45.554+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:45.693 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:45.398+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:45.694 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:45.398+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:45.694 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:45.398+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:45.694 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:45.399+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:46.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:45.689+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:46.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:45.689+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:46.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:45.689+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:46.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:45.689+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:46.681 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:46.408+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:46.682 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:46.408+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:46.682 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:46.408+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:46.682 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:46.408+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:46.682 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:46.603+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:46.682 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:46.603+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:46.682 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:46.603+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:46.682 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:46.603+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:47.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:46.677+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:47.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:46.677+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:47.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:46.677+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:47.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:46.678+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:47.704 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:47.594+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:47.704 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:47.594+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:47.704 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:47.594+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:47.704 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:47.594+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:47.704 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:47.433+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:47.704 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:47.433+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:47.704 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:47.433+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:47.704 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:47.433+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:48.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:47.699+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:48.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:47.699+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:48.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:47.699+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:48.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:47.701+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:48.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:48.477+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:48.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:48.477+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:48.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:48.477+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:48.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:48.477+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:48.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:48.691+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:48.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:48.691+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:48.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:48.691+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:48.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:48.691+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:48.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:48.602+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:48.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:48.602+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:48.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:48.602+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:48.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:48.602+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:49.624 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:07:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:07:49.361+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 1270 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:07:49.625 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:49.477+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:49.625 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:49.477+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:49.625 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:49.477+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:49.625 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:49.478+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:50.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:49.620+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:50.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:49.620+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:50.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:49.620+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:50.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:49.621+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:50.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:49.691+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:50.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:49.691+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:50.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:49.691+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:50.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:49.692+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:50.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:50.668+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:50.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:50.668+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:50.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:50.668+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:50.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:50.669+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:50.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:50.459+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:50.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:50.459+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:50.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:50.459+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:50.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:50.460+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:50.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:50.680+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:50.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:50.680+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:50.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:50.680+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:50.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:50.681+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:51.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:51.488+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:51.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:51.488+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:51.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:51.488+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:51.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:51.490+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:51.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:51.632+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:51.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:51.632+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:51.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:51.632+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:51.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:51.633+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:51.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:51.657+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:51.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:51.657+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:51.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:51.657+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:51.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:51.658+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:52.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:52.442+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:52.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:52.442+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:52.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:52.442+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:52.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:52.443+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:52.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:52.674+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:52.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:52.674+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:52.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:52.674+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:52.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:52.674+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:52.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:52.622+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:52.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:52.622+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:52.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:52.622+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:52.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:52.623+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:53.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:53.487+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:53.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:53.487+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:53.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:53.487+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:53.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:53.487+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:53.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:53.691+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:53.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:53.691+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:53.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:53.691+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:53.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:53.692+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:53.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:53.664+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:53.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:53.664+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:53.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:53.664+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:53.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:53.665+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:54.709 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:07:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:07:54.361+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 1364 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:07:54.710 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:54.444+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:54.710 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:54.444+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:54.710 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:54.444+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:54.710 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:54.445+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:55.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:54.712+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:55.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:54.712+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:55.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:54.712+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:55.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:54.713+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:55.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:54.705+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:55.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:54.705+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:55.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:54.705+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:55.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:54.706+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:55.711 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:55.438+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:55.711 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:55.438+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:55.711 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:55.438+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:55.711 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:55.439+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:56.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:55.757+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:56.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:55.757+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:56.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:55.757+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:56.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:55.758+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:56.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:55.707+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:56.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:55.707+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:56.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:55.707+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:56.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:55.708+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:56.723 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:56.438+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:56.723 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:56.438+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:56.723 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:56.438+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:56.723 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:56.438+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:57.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:56.718+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:57.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:56.719+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:57.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:56.719+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:57.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:56.719+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:57.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:56.761+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:57.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:56.761+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:57.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:56.761+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:57.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:56.762+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:57.751 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:57.419+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:57.751 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:57.419+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:57.751 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:57.419+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:57.751 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:57.419+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:58.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:57.747+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:58.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:57.747+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:58.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:57.747+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:58.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:57.748+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:58.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:57.751+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:58.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:57.751+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:58.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:57.751+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:58.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:57.751+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:58.712 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:58.424+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:58.713 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:58.424+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:58.713 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:58.424+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:58.713 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:58.425+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:59.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:58.761+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:59.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:58.761+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:59.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:58.761+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:07:59.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:58.762+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:59.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:58.708+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:59.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:58.708+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:59.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:58.708+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:07:59.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:58.709+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:07:59.737 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:07:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:07:59.362+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 1457 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:07:59.737 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:59.407+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:59.737 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:59.407+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:59.738 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:59.407+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:07:59.738 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:07:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:07:59.408+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:00.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:59.733+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:00.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:59.733+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:00.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:59.733+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:00.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:07:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:07:59.734+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:00.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:59.758+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:00.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:59.758+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:00.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:59.758+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:00.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:07:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:07:59.759+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:00.734 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:00.385+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:00.734 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:00.385+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:00.734 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:00.385+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:00.734 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:00.386+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:01.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:00.780+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:01.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:00.780+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:01.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:00.780+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:01.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:00.781+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:01.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:00.730+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:01.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:00.730+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:01.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:00.730+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:01.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:00.731+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:01.642 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:01.349+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:01.643 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:01.349+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:01.643 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:01.349+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:01.643 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:01.350+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:02.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:01.720+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:02.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:01.720+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:02.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:01.720+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:02.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:01.721+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:02.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:01.738+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:02.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:01.738+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:02.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:01.738+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:02.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:01.738+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:02.764 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:02.323+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:02.764 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:02.323+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:02.764 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:02.323+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:02.764 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:02.324+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:03.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:02.759+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:03.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:02.759+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:03.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:02.759+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:03.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:02.760+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:03.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:02.778+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:03.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:02.778+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:03.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:02.778+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:03.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:02.779+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:03.729 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:03.297+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:03.729 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:03.297+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:03.730 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:03.297+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:03.730 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:03.298+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:04.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:03.806+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:04.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:03.806+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:04.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:03.806+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:04.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:03.807+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:04.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:03.725+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:04.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:03.725+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:04.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:03.725+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:04.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:03.726+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:04.718 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:08:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:08:04.363+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 1553 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:08:04.718 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:04.343+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:04.718 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:04.343+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:04.719 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:04.343+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:04.719 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:04.343+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:05.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:04.712+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:05.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:04.712+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:05.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:04.712+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:05.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:04.713+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:05.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:04.830+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:05.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:04.830+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:05.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:04.830+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:05.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:04.830+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:05.674 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:05.353+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:05.675 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:05.353+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:05.675 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:05.353+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:05.675 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:05.354+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:06.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:05.832+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:06.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:05.832+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:06.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:05.832+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:06.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:05.833+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:06.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:05.670+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:06.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:05.670+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:06.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:05.670+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:06.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:05.671+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:06.681 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:06.371+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:06.681 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:06.371+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:06.682 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:06.371+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:06.682 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:06.371+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:07.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:06.677+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:07.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:06.677+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:07.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:06.677+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:07.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:06.677+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:07.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:06.864+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:07.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:06.864+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:07.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:06.864+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:07.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:06.864+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:07.724 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:07.405+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:07.724 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:07.405+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:07.724 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:07.405+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:07.724 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:07.406+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:07.975 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:07.719+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:07.975 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:07.720+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:07.975 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:07.720+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:07.975 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:07.720+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:07.975 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:07.900+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:07.975 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:07.900+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:07.975 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:07.900+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:07.976 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:07.901+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:08.771 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:08.403+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:08.771 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:08.403+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:08.771 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:08.403+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:08.772 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:08.404+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:09.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:08.938+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:09.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:08.938+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:09.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:08.938+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:09.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:08.939+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:09.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:08.767+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:09.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:08.767+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:09.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:08.767+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:09.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:08.768+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:09.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:08:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:08:09.363+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 1647 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:08:09.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:09.379+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:09.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:09.379+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:09.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:09.379+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:09.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:09.380+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:10.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:09.940+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:10.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:09.940+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:10.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:09.940+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:10.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:09.941+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:10.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:09.811+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:10.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:09.811+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:10.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:09.811+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:10.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:09.811+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:10.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:10.341+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:10.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:10.341+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:10.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:10.341+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:10.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:10.341+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:11.047 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:10.782+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:11.047 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:10.782+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:11.047 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:10.782+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:11.047 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:10.783+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:11.047 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:10.977+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:11.047 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:10.977+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:11.048 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:10.977+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:11.048 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:10.978+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:11.644 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:11.375+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:11.644 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:11.375+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:11.644 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:11.375+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:11.644 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:11.376+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:11.943 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:11.799+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:11.943 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:11.799+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:11.943 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:11.799+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:11.943 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:11.800+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:12.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:11.939+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:12.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:11.939+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:12.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:11.939+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:12.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:11.940+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:12.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:12.382+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:12.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:12.382+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:12.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:12.382+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:12.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:12.383+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:13.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:12.913+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:13.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:12.913+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:13.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:12.913+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:13.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:12.914+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:13.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:12.782+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:13.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:12.782+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:13.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:12.782+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:13.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:12.783+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:13.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:13.373+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:13.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:13.373+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:13.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:13.373+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:13.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:13.374+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:14.119 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:13.937+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:14.119 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:13.937+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:14.119 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:13.937+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:14.119 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:13.938+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:14.119 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:13.779+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:14.120 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:13.779+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:14.120 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:13.779+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:14.120 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:13.780+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:14.373 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:08:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:08:14.364+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 1742 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:08:14.752 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:14.368+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:14.752 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:14.368+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:14.752 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:14.368+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:14.752 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:14.370+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:15.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:14.748+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:15.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:14.748+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:15.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:14.748+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:15.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:14.748+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:15.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:14.950+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:15.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:14.950+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:15.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:14.950+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:15.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:14.951+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:15.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:15.372+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:15.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:15.372+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:15.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:15.372+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:15.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:15.372+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:16.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:15.792+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:16.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:15.792+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:16.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:15.792+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:16.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:15.793+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:16.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:15.947+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:16.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:15.947+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:16.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:15.947+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:16.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:15.948+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:16.762 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:16.330+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:16.763 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:16.330+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:16.763 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:16.330+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:16.763 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:16.332+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:17.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:16.959+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:17.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:16.959+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:17.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:16.959+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:17.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:16.960+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:17.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:16.759+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:17.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:16.759+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:17.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:16.759+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:17.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:16.759+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:17.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:17.379+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:17.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:17.379+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:17.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:17.379+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:17.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:17.379+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:18.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:17.977+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:18.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:17.977+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:18.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:17.977+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:18.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:17.978+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:18.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:17.729+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:18.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:17.729+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:18.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:17.729+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:18.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:17.730+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:18.750 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:18.347+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:18.750 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:18.347+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:18.750 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:18.347+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:18.750 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:18.347+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:19.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:18.952+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:19.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:18.952+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:19.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:18.952+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:19.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:18.953+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:19.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:18.747+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:19.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:18.747+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:19.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:18.747+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:19.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:18.747+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:19.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:08:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:08:19.365+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 1835 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:08:19.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:19.348+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:19.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:19.348+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:19.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:19.348+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:19.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:19.349+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:20.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:19.949+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:20.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:19.949+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:20.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:19.949+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:20.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:19.950+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:20.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:19.780+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:20.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:19.780+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:20.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:19.780+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:20.282 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:19.781+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:20.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:20.337+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:20.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:20.337+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:20.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:20.337+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:20.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:20.338+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:21.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:20.919+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:21.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:20.919+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:21.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:20.919+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:21.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:20.920+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:21.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:20.781+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:21.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:20.781+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:21.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:20.781+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:21.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:20.782+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:21.644 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:21.290+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:21.644 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:21.290+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:21.644 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:21.290+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:21.645 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:21.291+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:22.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:21.884+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:22.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:21.884+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:22.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:21.884+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:22.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:21.885+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:22.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:21.751+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:22.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:21.751+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:22.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:21.751+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:22.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:21.751+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:22.717 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:22.309+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:22.718 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:22.309+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:22.718 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:22.309+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:22.718 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:22.310+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:23.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:22.843+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:23.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:22.843+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:23.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:22.843+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:23.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:22.844+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:23.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:22.713+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:23.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:22.713+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:23.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:22.713+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:23.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:22.714+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:23.689 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:23.311+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:23.689 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:23.311+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:23.689 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:23.311+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:23.689 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:23.312+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:24.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:23.806+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:24.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:23.806+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:24.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:23.806+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:24.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:23.806+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:24.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:23.684+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:24.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:23.684+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:24.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:23.684+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:24.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:23.685+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:24.685 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:08:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:08:24.365+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 1931 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:08:24.685 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:24.359+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:24.685 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:24.359+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:24.686 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:24.359+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:24.686 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:24.360+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:25.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:24.681+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:25.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:24.681+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:25.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:24.681+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:25.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:24.682+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:25.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:24.818+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:25.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:24.818+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:25.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:24.818+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:25.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:24.818+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:25.679 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:25.346+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:25.679 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:25.346+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:25.679 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:25.346+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:25.679 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:25.347+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:26.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:25.776+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:26.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:25.776+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:26.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:25.776+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:26.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:25.777+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:26.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:25.675+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:26.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:25.675+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:26.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:25.675+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:26.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:25.676+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:26.668 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:26.307+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:26.668 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:26.307+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:26.668 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:26.307+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:26.668 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:26.308+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:27.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:26.664+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:27.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:26.664+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:27.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:26.664+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:27.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:26.664+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:27.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:26.769+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:27.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:26.769+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:27.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:26.769+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:27.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:26.770+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:27.707 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:27.304+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:27.707 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:27.304+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:27.708 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:27.304+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:27.708 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:27.304+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:28.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:27.703+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:28.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:27.703+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:28.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:27.703+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:28.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:27.704+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:28.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:27.758+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:28.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:27.758+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:28.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:27.758+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:28.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:27.758+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:28.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:28.260+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:28.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:28.260+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:28.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:28.260+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:28.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:28.261+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:29.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:28.784+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:29.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:28.784+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:29.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:28.784+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:29.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:28.785+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:29.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:28.664+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:29.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:28.664+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:29.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:28.664+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:29.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:28.665+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:29.688 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:08:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:08:29.366+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 2025 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:08:29.688 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:29.285+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:29.688 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:29.285+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:29.688 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:29.285+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:29.688 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:29.286+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:30.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:29.766+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:30.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:29.766+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:30.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:29.766+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:30.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:29.767+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:30.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:29.683+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:30.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:29.683+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:30.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:29.683+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:30.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:29.685+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:30.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:30.255+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:30.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:30.255+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:30.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:30.255+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:30.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:30.256+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:31.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:30.719+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:31.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:30.719+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:31.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:30.719+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:31.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:30.720+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:31.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:30.685+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:31.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:30.685+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:31.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:30.685+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:31.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:30.685+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:31.645 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:31.284+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:31.645 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:31.284+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:31.645 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:31.284+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:31.645 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:31.284+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:32.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:31.759+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:32.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:31.759+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:32.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:31.759+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:32.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:31.760+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:32.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:31.679+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:32.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:31.679+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:32.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:31.679+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:32.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:31.680+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:32.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:32.275+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:32.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:32.275+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:32.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:32.275+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:32.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:32.276+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:33.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:32.742+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:33.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:32.742+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:33.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:32.742+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:33.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:32.744+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:33.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:32.635+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:33.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:32.635+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:33.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:32.635+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:33.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:32.636+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:33.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:33.235+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:33.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:33.235+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:33.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:33.235+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:33.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:33.236+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:34.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:33.779+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:34.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:33.779+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:34.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:33.779+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:34.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:33.780+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:34.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:33.654+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:34.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:33.654+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:34.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:33.654+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:34.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:33.655+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:34.701 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:08:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:08:34.367+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 2121 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:08:34.701 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:34.282+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:34.701 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:34.282+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:34.701 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:34.282+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:34.701 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:34.283+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:35.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:34.812+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:35.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:34.812+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:35.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:34.812+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:35.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:34.813+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:35.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:34.697+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:35.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:34.697+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:35.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:34.697+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:35.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:34.698+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:35.622 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:35.330+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:35.622 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:35.330+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:35.622 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:35.330+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:35.622 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:35.330+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:36.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:35.784+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:36.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:35.784+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:36.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:35.784+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:36.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:35.785+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:36.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:35.728+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:36.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:35.728+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:36.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:35.728+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:36.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:35.728+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:36.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:36.358+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:36.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:36.358+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:36.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:36.358+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:36.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:36.358+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:37.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:36.816+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:37.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:36.816+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:37.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:36.816+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:37.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:36.817+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:37.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:36.775+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:37.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:36.775+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:37.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:36.775+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:37.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:36.776+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:37.749 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:37.356+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:37.749 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:37.356+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:37.749 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:37.356+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:37.749 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:37.357+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:38.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:37.745+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:38.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:37.745+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:38.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:37.745+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:38.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:37.746+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:38.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:37.802+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:38.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:37.802+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:38.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:37.802+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:38.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:37.803+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:38.716 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:38.341+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:38.716 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:38.341+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:38.716 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:38.341+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:38.716 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:38.342+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:39.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:38.818+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:39.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:38.818+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:39.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:38.818+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:39.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:38.819+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:39.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:38.711+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:39.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:38.711+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:39.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:38.711+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:39.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:38.713+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:39.701 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:08:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:08:39.367+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 2214 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:08:39.702 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:39.303+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:39.702 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:39.303+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:39.702 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:39.303+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:39.702 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:39.304+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:40.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:39.697+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:40.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:39.697+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:40.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:39.697+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:40.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:39.698+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:40.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:39.855+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:40.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:39.855+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:40.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:39.855+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:40.032 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:39.855+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:40.705 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:40.277+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:40.706 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:40.277+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:40.706 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:40.277+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:40.706 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:40.278+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:41.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:40.700+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:41.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:40.700+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:41.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:40.700+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:41.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:40.701+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:41.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:40.864+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:41.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:40.864+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:41.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:40.864+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:41.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:40.864+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:41.646 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:41.321+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:41.646 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:41.321+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:41.646 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:41.321+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:41.646 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:41.322+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:42.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:41.713+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:42.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:41.713+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:42.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:41.713+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:42.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:41.714+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:42.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:41.891+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:42.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:41.891+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:42.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:41.891+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:42.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:41.892+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:42.686 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:42.362+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:42.687 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:42.362+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:42.687 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:42.362+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:42.687 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:42.363+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:43.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:42.682+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:43.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:42.682+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:43.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:42.682+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:43.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:42.683+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:43.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:42.892+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:43.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:42.892+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:43.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:42.892+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:43.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:42.893+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:43.737 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:43.341+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:43.737 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:43.341+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:43.737 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:43.341+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:43.737 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:43.341+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:44.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:43.733+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:44.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:43.733+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:44.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:43.733+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:44.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:43.734+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:44.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:43.921+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:44.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:43.921+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:44.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:43.921+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:44.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:43.922+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:44.775 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:08:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:08:44.368+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 2311 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:08:44.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:44.380+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:44.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:44.380+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:44.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:44.380+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:44.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:44.381+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:45.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:44.927+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:45.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:44.927+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:45.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:44.927+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:45.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:44.928+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:45.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:44.771+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:45.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:44.771+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:45.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:44.771+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:45.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:44.772+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:45.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:45.373+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:45.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:45.373+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:45.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:45.373+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:45.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:45.373+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:46.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:45.956+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:46.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:45.956+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:46.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:45.956+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:46.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:45.957+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:46.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:45.806+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:46.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:45.806+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:46.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:45.806+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:46.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:45.807+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:46.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:46.410+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:46.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:46.410+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:46.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:46.410+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:46.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:46.411+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:47.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:47.002+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:47.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:47.002+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:47.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:47.002+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:47.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:47.003+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:47.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:46.852+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:47.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:46.852+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:47.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:46.852+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:47.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:46.853+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:47.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:47.376+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:47.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:47.376+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:47.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:47.376+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:47.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:47.377+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:48.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:47.958+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:48.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:47.958+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:48.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:47.958+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:48.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:47.959+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:48.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:47.888+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:48.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:47.888+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:48.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:47.888+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:48.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:47.889+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:48.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:48.403+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:48.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:48.403+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:48.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:48.403+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:48.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:48.404+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:49.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:48.953+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:49.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:48.953+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:49.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:48.953+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:49.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:48.954+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:49.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:48.905+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:49.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:48.905+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:49.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:48.905+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:49.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:48.906+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:49.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:08:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:08:49.369+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 2405 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:08:49.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:49.420+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:49.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:49.420+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:49.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:49.420+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:49.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:49.420+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:50.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:49.948+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:50.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:49.948+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:50.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:49.948+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:50.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:49.949+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:50.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:50.002+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:50.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:50.002+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:50.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:50.002+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:50.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:50.003+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:50.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:50.385+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:50.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:50.385+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:50.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:50.385+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:50.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:50.386+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:51.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:50.963+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:51.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:50.963+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:51.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:50.963+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:51.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:50.964+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:51.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:51.001+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:51.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:51.001+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:51.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:51.001+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:51.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:51.001+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:51.647 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:51.380+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:51.647 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:51.380+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:51.647 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:51.380+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:51.647 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:51.381+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:52.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:51.973+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:52.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:51.973+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:52.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:51.973+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:52.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:51.974+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:52.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:51.964+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:52.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:51.964+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:52.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:51.964+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:52.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:51.965+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:52.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:52.397+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:52.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:52.397+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:52.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:52.397+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:52.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:52.398+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:53.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:53.014+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:53.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:53.014+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:53.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:53.014+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:53.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:53.014+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:53.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:52.928+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:53.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:52.928+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:53.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:52.928+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:53.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:52.929+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:53.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:53.395+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:53.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:53.395+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:53.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:53.395+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:53.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:53.396+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:54.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:53.967+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:54.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:53.967+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:54.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:53.967+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:54.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:53.968+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:54.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:53.929+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:54.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:53.929+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:54.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:53.929+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:54.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:53.930+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:54.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:08:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:08:54.369+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 2501 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:08:54.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:54.385+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:54.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:54.385+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:54.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:54.385+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:54.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:54.386+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:55.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:55.009+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:55.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:55.009+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:55.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:55.009+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:55.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:55.010+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:55.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:54.894+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:55.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:54.894+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:55.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:54.894+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:55.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:54.895+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:55.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:55.426+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:55.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:55.426+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:55.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:55.426+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:55.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:55.427+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:56.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:55.963+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:56.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:55.963+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:56.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:55.963+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:56.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:55.964+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:56.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:55.908+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:56.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:55.908+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:56.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:55.908+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:56.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:55.909+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:56.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:56.414+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:56.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:56.414+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:56.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:56.414+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:56.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:56.414+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:57.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:56.962+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:57.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:56.962+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:57.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:56.962+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:57.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:56.963+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:57.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:56.919+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:57.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:56.919+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:57.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:56.919+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:57.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:56.920+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:57.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:57.368+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:57.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:57.368+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:57.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:57.368+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:57.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:57.370+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:58.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:57.941+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:58.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:57.941+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:58.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:57.941+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:58.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:57.942+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:58.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:57.960+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:58.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:57.960+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:58.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:57.960+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:58.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:57.961+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:58.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:58.394+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:58.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:58.394+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:58.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:58.394+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:58.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:58.395+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:59.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:58.910+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:59.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:58.910+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:59.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:58.910+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:08:59.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:58.911+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:59.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:58.987+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:59.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:58.987+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:59.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:58.987+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:08:59.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:58.988+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:08:59.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:08:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:08:59.370+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 2595 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:08:59.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:59.440+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:59.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:59.440+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:59.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:59.440+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:08:59.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:08:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:08:59.441+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:00.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:59.970+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:00.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:59.970+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:00.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:59.970+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:00.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:08:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:08:59.971+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:00.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:59.928+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:00.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:59.928+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:00.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:59.928+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:00.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:08:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:08:59.929+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:00.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:00.408+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:00.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:00.408+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:00.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:00.408+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:00.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:00.409+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:01.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:00.943+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:01.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:00.943+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:01.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:00.943+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:01.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:00.944+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:01.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:00.952+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:01.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:00.952+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:01.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:00.952+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:01.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:00.952+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:01.648 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:01.378+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:01.648 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:01.378+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:01.648 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:01.378+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:01.648 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:01.378+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:02.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:01.970+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:02.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:01.970+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:02.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:01.970+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:02.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:01.971+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:02.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:01.931+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:02.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:01.931+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:02.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:01.931+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:02.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:01.932+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:02.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:02.401+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:02.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:02.401+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:02.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:02.401+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:02.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:02.402+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:03.270 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:02.911+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:03.270 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:02.911+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:03.270 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:02.911+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:03.270 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:02.912+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:03.271 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:02.959+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:03.271 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:02.959+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:03.271 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:02.959+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:03.271 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:02.960+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:03.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:03.438+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:03.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:03.438+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:03.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:03.438+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:03.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:03.439+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:04.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:03.978+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:04.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:03.978+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:04.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:03.978+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:04.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:03.979+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:04.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:03.899+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:04.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:03.899+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:04.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:03.899+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:04.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:03.900+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:04.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:09:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:09:04.371+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 2692 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:09:04.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:04.469+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:04.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:04.469+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:04.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:04.469+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:04.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:04.470+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:05.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:04.941+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:05.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:04.941+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:05.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:04.941+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:05.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:04.942+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:05.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:04.990+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:05.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:04.990+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:05.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:04.990+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:05.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:04.991+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:05.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:05.427+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:05.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:05.427+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:05.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:05.427+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:05.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:05.428+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:06.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:05.968+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:06.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:05.968+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:06.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:05.968+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:06.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:05.969+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:06.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:05.978+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:06.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:05.978+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:06.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:05.978+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:06.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:05.979+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:06.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:06.437+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:06.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:06.437+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:06.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:06.437+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:06.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:06.438+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:07.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:06.992+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:07.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:06.992+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:07.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:06.992+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:07.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:06.993+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:07.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:07.021+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:07.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:07.021+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:07.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:07.021+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:07.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:07.022+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:07.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:07.451+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:07.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:07.451+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:07.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:07.451+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:07.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:07.452+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:08.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:08.023+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:08.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:08.023+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:08.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:08.023+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:08.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:08.024+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:08.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:08.000+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:08.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:08.000+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:08.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:08.000+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:08.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:08.000+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:08.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:08.466+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:08.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:08.467+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:08.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:08.467+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:08.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:08.468+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:09.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:09.006+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:09.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:09.006+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:09.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:09.006+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:09.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:09.006+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:09.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:09.004+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:09.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:09.004+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:09.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:09.004+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:09.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:09.005+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:09.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:09:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:09:09.371+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 2787 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:09:09.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:09.495+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:09.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:09.495+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:09.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:09.495+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:09.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:09.496+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:10.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:10.053+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:10.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:10.053+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:10.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:10.053+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:10.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:10.054+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:10.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:10.017+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:10.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:10.017+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:10.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:10.017+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:10.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:10.018+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:10.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:10.519+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:10.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:10.519+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:10.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:10.519+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:10.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:10.520+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:11.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:11.054+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:11.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:11.054+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:11.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:11.054+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:11.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:11.055+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:11.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:11.086+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:11.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:11.086+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:11.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:11.086+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:11.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:11.086+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:12.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:11.569+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:12.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:11.569+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:12.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:11.569+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:12.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:11.570+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:12.486 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:12.097+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:12.486 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:12.097+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:12.486 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:12.097+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:12.486 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:12.098+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:12.486 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:12.046+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:12.487 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:12.046+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:12.487 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:12.046+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:12.487 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:12.047+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:12.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:12.617+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:12.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:12.617+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:12.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:12.617+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:12.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:12.618+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:13.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:13.119+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:13.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:13.119+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:13.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:13.119+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:13.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:13.120+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:13.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:13.096+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:13.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:13.096+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:13.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:13.096+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:13.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:13.097+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:14.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:13.660+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:14.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:13.661+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:14.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:13.661+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:14.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:13.662+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:14.376 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:14.118+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:14.377 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:14.118+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:14.377 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:14.118+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:14.377 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:14.119+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:14.377 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:14.125+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:14.377 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:14.125+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:14.377 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:14.125+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:14.377 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:14.126+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:14.629 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:09:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:09:14.372+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 2883 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:09:15.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:14.624+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:15.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:14.624+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:15.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:14.624+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:15.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:14.626+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:15.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:15.110+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:15.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:15.110+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:15.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:15.110+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:15.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:15.111+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:15.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:15.092+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:15.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:15.092+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:15.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:15.092+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:15.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:15.093+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:16.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:15.656+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:16.031 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:15.656+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:16.031 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:15.656+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:16.031 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:15.656+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:16.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:16.116+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:16.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:16.116+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:16.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:16.116+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:16.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:16.117+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:16.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:16.098+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:16.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:16.098+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:16.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:16.098+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:16.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:16.100+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:17.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:16.627+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:17.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:16.627+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:17.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:16.627+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:17.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:16.628+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:17.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:17.125+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:17.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:17.125+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:17.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:17.125+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:17.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:17.125+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:17.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:17.139+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:17.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:17.139+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:17.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:17.139+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:17.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:17.139+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:18.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:17.676+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:18.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:17.676+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:18.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:17.676+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:18.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:17.676+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:18.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:18.156+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:18.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:18.156+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:18.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:18.156+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:18.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:18.157+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:18.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:18.125+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:18.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:18.125+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:18.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:18.125+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:18.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:18.126+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:19.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:18.682+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:19.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:18.682+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:19.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:18.682+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:19.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:18.682+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:19.505 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:09:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:09:19.374+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 2976 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:09:19.505 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:19.203+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:19.506 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:19.203+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:19.506 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:19.203+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:19.506 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:19.203+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:19.506 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:19.161+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:19.506 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:19.161+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:19.506 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:19.161+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:19.506 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:19.161+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:19.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:19.726+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:19.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:19.726+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:19.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:19.726+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:19.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:19.726+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:20.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:20.173+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:20.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:20.173+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:20.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:20.173+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:20.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:20.174+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:20.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:20.145+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:20.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:20.145+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:20.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:20.145+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:20.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:20.146+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:21.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:20.749+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:21.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:20.749+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:21.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:20.749+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:21.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:20.750+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:21.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:21.213+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:21.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:21.213+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:21.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:21.213+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:21.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:21.214+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:21.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:21.174+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:21.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:21.174+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:21.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:21.174+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:21.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:21.176+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:22.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:21.712+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:22.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:21.712+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:22.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:21.712+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:22.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:21.713+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:22.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:22.196+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:22.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:22.196+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:22.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:22.196+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:22.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:22.197+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:22.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:22.184+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:22.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:22.184+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:22.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:22.184+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:22.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:22.184+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:23.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:22.708+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:23.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:22.708+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:23.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:22.708+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:23.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:22.709+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:23.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:23.207+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:23.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:23.207+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:23.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:23.207+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:23.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:23.208+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:23.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:23.154+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:23.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:23.154+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:23.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:23.154+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:23.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:23.155+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:24.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:23.737+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:24.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:23.737+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:24.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:23.737+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:24.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:23.738+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:24.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:09:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:09:24.374+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 3072 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:09:24.503 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:24.241+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:24.503 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:24.241+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:24.503 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:24.241+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:24.503 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:24.242+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:24.503 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:24.174+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:24.503 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:24.174+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:24.503 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:24.174+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:24.503 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:24.175+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:24.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:24.742+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:24.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:24.742+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:24.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:24.742+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:24.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:24.742+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:25.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:25.264+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:25.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:25.264+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:25.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:25.264+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:25.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:25.265+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:25.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:25.184+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:25.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:25.184+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:25.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:25.184+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:25.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:25.185+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:26.141 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:25.776+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:26.142 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:25.776+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:26.142 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:25.776+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:26.142 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:25.778+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:26.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:26.137+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:26.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:26.137+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:26.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:26.137+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:26.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:26.138+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:26.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:26.312+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:26.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:26.312+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:26.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:26.312+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:26.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:26.313+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:27.103 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:26.784+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:27.104 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:26.784+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:27.104 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:26.784+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:27.104 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:26.785+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:27.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:27.297+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:27.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:27.297+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:27.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:27.297+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:27.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:27.298+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:27.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:27.099+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:27.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:27.099+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:27.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:27.099+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:27.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:27.100+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:28.274 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:27.831+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:28.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:27.831+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:28.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:27.831+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:28.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:27.832+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:28.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:28.071+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:28.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:28.071+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:28.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:28.071+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:28.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:28.072+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:28.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:28.270+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:28.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:28.270+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:28.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:28.270+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:28.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:28.271+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:29.229 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:28.854+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:29.229 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:28.854+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:29.230 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:28.854+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:29.230 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:28.855+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:29.230 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:29.030+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:29.230 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:29.030+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:29.230 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:29.030+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:29.230 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:29.030+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:29.504 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:09:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:09:29.374+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 3167 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:09:29.504 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:29.225+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:29.504 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:29.225+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:29.504 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:29.225+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:29.504 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:29.226+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:30.232 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:30.054+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:30.232 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:30.054+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:30.232 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:30.054+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:30.232 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:30.054+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:30.232 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:29.884+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:30.232 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:29.884+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:30.232 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:29.884+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:30.232 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:29.884+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:30.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:30.228+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:30.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:30.228+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:30.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:30.228+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:30.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:30.228+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:31.242 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:30.924+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:31.242 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:30.924+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:31.243 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:30.924+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:31.243 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:30.925+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:31.243 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:31.008+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:31.243 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:31.008+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:31.243 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:31.008+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:31.243 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:31.009+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:31.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:31.238+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:31.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:31.238+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:31.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:31.238+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:31.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:31.238+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:32.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:31.891+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:32.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:31.891+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:32.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:31.891+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:32.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:31.891+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:32.289 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:32.028+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:32.290 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:32.028+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:32.290 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:32.028+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:32.290 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:32.029+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:32.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:32.285+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:32.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:32.285+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:32.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:32.285+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:32.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:32.286+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:33.269 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:32.880+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:33.269 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:32.880+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:33.269 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:32.880+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:33.269 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:32.881+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:33.270 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:32.984+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:33.270 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:32.984+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:33.270 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:32.984+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:33.270 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:32.985+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:33.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:33.265+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:33.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:33.265+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:33.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:33.265+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:33.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:33.266+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:34.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:33.991+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:34.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:33.991+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:34.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:33.991+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:34.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:33.992+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:34.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:33.879+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:34.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:33.879+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:34.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:33.879+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:34.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:33.880+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:34.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:09:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:09:34.375+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 3262 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:09:34.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:34.305+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:34.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:34.305+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:34.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:34.305+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:34.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:34.306+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:35.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:34.914+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:35.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:34.914+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:35.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:34.914+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:35.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:34.915+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:35.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:34.954+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:35.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:34.954+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:35.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:34.954+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:35.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:34.955+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:35.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:35.288+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:35.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:35.288+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:35.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:35.288+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:35.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:35.288+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:36.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:35.888+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:36.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:35.888+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:36.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:35.888+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:36.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:35.889+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:36.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:35.963+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:36.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:35.963+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:36.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:35.963+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:36.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:35.964+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:36.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:36.312+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:36.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:36.312+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:36.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:36.312+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:36.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:36.313+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:37.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:36.895+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:37.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:36.895+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:37.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:36.895+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:37.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:36.896+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:37.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:36.960+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:37.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:36.960+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:37.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:36.960+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:37.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:36.960+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:37.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:37.274+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:37.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:37.274+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:37.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:37.274+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:37.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:37.275+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:38.253 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:37.934+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:38.253 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:37.934+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:38.253 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:37.934+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:38.253 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:37.935+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:38.253 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:37.871+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:38.253 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:37.871+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:38.253 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:37.871+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:38.253 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:37.872+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:38.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:38.249+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:38.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:38.249+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:38.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:38.249+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:38.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:38.250+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:39.206 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:38.935+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:39.206 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:38.935+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:39.207 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:38.935+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:39.207 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:38.936+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:39.207 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:38.826+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:39.207 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:38.826+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:39.207 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:38.826+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:39.207 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:38.827+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:39.505 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:09:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:09:39.376+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 3356 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:09:39.505 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:39.202+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:39.505 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:39.202+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:39.505 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:39.202+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:39.505 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:39.203+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:40.212 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:39.780+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:40.212 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:39.780+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:40.212 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:39.780+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:40.212 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:39.781+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:40.213 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:39.984+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:40.213 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:39.984+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:40.213 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:39.984+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:40.213 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:39.985+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:40.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:40.208+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:40.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:40.208+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:40.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:40.208+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:40.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:40.209+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:41.198 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:40.800+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:41.198 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:40.800+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:41.199 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:40.800+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:41.199 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:40.801+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:41.199 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:40.946+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:41.199 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:40.946+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:41.199 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:40.946+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:41.199 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:40.947+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:41.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:41.194+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:41.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:41.194+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:41.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:41.194+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:41.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:41.195+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:41.938 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:41.803+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:41.938 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:41.803+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:41.938 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:41.803+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:41.938 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:41.804+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:42.244 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:41.934+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:42.244 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:41.934+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:42.244 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:41.934+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:42.244 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:41.935+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:42.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:42.240+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:42.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:42.240+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:42.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:42.240+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:42.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:42.241+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:43.206 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:42.836+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:43.206 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:42.836+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:43.206 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:42.836+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:43.206 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:42.837+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:43.206 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:42.957+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:43.206 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:42.957+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:43.207 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:42.957+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:43.207 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:42.958+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:43.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:43.274+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:43.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:43.274+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:43.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:43.274+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:43.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:43.274+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:44.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:43.999+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:44.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:43.999+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:44.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:43.999+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:44.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:44.000+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:44.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:43.839+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:44.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:43.839+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:44.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:43.839+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:44.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:43.840+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:44.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:09:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:09:44.376+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 3450 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:09:44.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:44.311+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:44.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:44.311+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:44.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:44.311+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:44.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:44.312+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:45.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:44.825+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:45.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:44.825+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:45.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:44.825+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:45.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:44.826+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:45.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:45.027+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:45.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:45.027+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:45.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:45.027+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:45.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:45.027+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:45.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:45.343+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:45.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:45.343+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:45.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:45.343+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:45.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:45.344+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:46.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:46.020+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:46.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:46.020+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:46.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:46.020+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:46.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:46.021+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:46.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:45.812+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:46.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:45.812+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:46.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:45.812+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:46.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:45.813+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:46.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:46.378+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:46.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:46.378+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:46.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:46.378+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:46.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:46.378+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:47.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:47.069+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:47.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:47.069+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:47.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:47.069+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:47.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:47.070+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:47.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:46.848+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:47.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:46.848+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:47.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:46.848+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:47.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:46.848+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:47.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:47.371+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:47.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:47.371+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:47.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:47.371+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:47.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:47.372+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:48.106 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:47.840+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:48.106 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:47.840+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:48.106 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:47.840+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:48.106 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:47.840+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:48.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:48.102+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:48.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:48.102+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:48.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:48.102+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:48.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:48.103+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:48.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:48.346+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:48.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:48.346+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:48.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:48.346+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:48.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:48.347+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:49.125 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:48.824+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:49.126 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:48.824+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:49.126 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:48.824+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:49.126 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:48.825+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:49.381 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:49.330+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:49.382 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:49.330+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:49.382 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:49.330+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:49.382 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:49.330+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:49.382 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:49.121+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:49.382 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:49.121+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:49.382 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:49.121+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:49.382 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:49.122+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:49.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:09:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:09:49.377+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 3545 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:09:50.164 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:49.866+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:50.164 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:49.866+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:50.164 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:49.866+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:50.164 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:49.867+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:50.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:50.160+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:50.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:50.160+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:50.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:50.160+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:50.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:50.160+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:50.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:50.330+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:50.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:50.330+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:50.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:50.330+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:50.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:50.331+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:51.199 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:50.860+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:51.199 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:50.860+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:51.199 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:50.860+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:51.199 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:50.861+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:51.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:51.298+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:51.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:51.298+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:51.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:51.298+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:51.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:51.299+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:51.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:51.194+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:51.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:51.195+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:51.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:51.195+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:51.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:51.196+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:52.183 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:51.909+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:52.183 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:51.909+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:52.183 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:51.909+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:52.183 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:51.910+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:52.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:52.179+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:52.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:52.179+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:52.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:52.179+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:52.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:52.180+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:52.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:52.331+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:52.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:52.331+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:52.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:52.331+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:52.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:52.332+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:53.196 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:52.874+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:53.196 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:52.874+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:53.196 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:52.874+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:53.196 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:52.874+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:53.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:53.192+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:53.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:53.192+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:53.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:53.192+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:53.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:53.193+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:53.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:53.369+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:53.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:53.369+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:53.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:53.369+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:53.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:53.370+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:54.165 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:53.867+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:54.165 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:53.867+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:54.165 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:53.867+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:54.166 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:53.867+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:54.505 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:09:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:09:54.378+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 3639 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:09:54.506 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:54.161+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:54.506 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:54.161+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:54.506 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:54.161+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:54.506 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:54.162+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:54.506 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:54.365+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:54.506 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:54.365+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:54.506 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:54.365+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:54.507 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:54.366+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:55.199 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:54.820+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:55.199 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:54.820+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:55.199 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:54.820+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:55.199 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:54.820+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:55.494 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:55.194+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:55.494 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:55.194+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:55.494 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:55.194+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:55.494 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:55.195+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:55.495 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:55.376+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:55.495 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:55.376+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:55.495 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:55.376+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:55.495 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:55.377+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:56.221 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:55.782+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:56.221 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:55.782+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:56.221 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:55.782+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:56.221 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:55.782+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:56.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:56.404+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:56.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:56.404+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:56.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:56.404+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:56.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:56.404+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:56.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:56.217+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:56.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:56.217+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:56.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:56.217+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:56.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:56.218+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:57.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:56.759+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:57.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:56.759+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:57.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:56.759+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:57.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:56.759+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:57.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:57.174+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:57.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:57.174+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:57.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:57.174+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:57.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:57.174+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:57.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:57.381+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:57.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:57.381+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:57.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:57.381+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:57.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:57.382+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:58.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:57.748+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:58.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:57.748+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:58.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:57.748+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:58.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:57.749+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:58.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:58.381+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:58.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:58.381+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:58.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:58.381+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:58.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:58.381+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:58.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:58.194+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:58.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:58.194+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:58.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:58.194+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:58.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:58.195+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:59.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:58.732+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:59.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:58.732+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:59.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:58.732+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:59.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:58.733+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:59.508 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:09:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:09:59.379+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 3733 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:09:59.508 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:59.376+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:59.508 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:59.376+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:59.509 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:59.376+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:09:59.509 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:09:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:09:59.377+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:59.509 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:59.196+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:59.509 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:59.196+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:59.509 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:59.196+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:09:59.509 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:09:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:09:59.197+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:09:59.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:59.729+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:59.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:59.729+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:59.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:59.729+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:09:59.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:09:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:09:59.730+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:00.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:00.389+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:00.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:00.389+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:00.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:00.389+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:00.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:00.390+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:00.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:00.190+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:00.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:00.190+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:00.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:00.190+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:00.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:00.191+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:01.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:00.762+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:01.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:00.762+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:01.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:00.762+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:01.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:00.763+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:01.433 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:01.143+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:01.433 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:01.143+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:01.433 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:01.143+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:01.433 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:01.144+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:01.721 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:01.429+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:01.721 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:01.429+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:01.721 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:01.429+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:01.722 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:01.430+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:02.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:01.717+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:02.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:01.717+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:02.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:01.717+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:02.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:01.718+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:02.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:02.168+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:02.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:02.168+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:02.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:02.168+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:02.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:02.168+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:02.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:02.404+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:02.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:02.404+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:02.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:02.404+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:02.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:02.405+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:03.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:02.760+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:03.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:02.760+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:03.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:02.760+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:03.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:02.761+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:03.401 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:03.125+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:03.401 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:03.125+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:03.402 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:03.125+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:03.402 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:03.126+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:03.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:03.397+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:03.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:03.397+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:03.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:03.397+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:03.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:03.398+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:04.162 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:03.783+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:04.163 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:03.783+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:04.163 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:03.783+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:04.163 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:03.784+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:04.450 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:04.158+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:04.450 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:04.158+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:04.450 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:04.158+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:04.450 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:04.159+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:04.451 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:10:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:10:04.379+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 3828 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:10:04.710 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:04.446+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:04.710 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:04.446+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:04.710 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:04.446+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:04.710 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:04.447+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:05.029 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:04.833+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:05.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:04.833+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:05.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:04.833+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:05.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:04.834+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:05.429 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:05.139+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:05.429 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:05.139+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:05.429 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:05.139+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:05.429 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:05.140+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:05.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:05.425+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:05.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:05.425+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:05.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:05.425+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:05.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:05.426+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:06.144 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:05.862+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:06.144 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:05.862+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:06.145 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:05.862+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:06.145 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:05.863+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:06.443 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:06.140+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:06.443 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:06.140+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:06.443 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:06.140+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:06.443 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:06.141+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:06.779 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:06.439+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:06.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:06.439+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:06.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:06.439+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:06.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:06.439+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:07.116 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:06.820+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:07.116 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:06.820+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:07.116 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:06.820+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:07.117 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:06.821+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:07.460 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:07.112+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:07.461 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:07.112+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:07.461 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:07.112+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:07.461 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:07.113+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:07.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:07.456+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:07.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:07.456+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:07.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:07.456+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:07.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:07.457+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:08.162 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:07.796+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:08.162 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:07.796+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:08.162 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:07.796+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:08.162 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:07.797+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:08.436 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:08.158+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:08.436 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:08.158+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:08.437 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:08.158+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:08.437 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:08.158+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:08.779 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:08.432+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:08.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:08.432+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:08.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:08.432+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:08.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:08.433+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:09.185 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:08.812+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:09.185 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:08.812+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:09.185 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:08.812+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:09.185 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:08.812+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:09.509 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:10:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:10:09.380+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 3922 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:10:09.509 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:09.181+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:09.510 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:09.181+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:09.510 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:09.181+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:09.510 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:09.182+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:09.510 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:09.396+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:09.510 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:09.396+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:09.510 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:09.396+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:09.510 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:09.396+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:10.189 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:09.854+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:10.189 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:09.854+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:10.189 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:09.854+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:10.189 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:09.854+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:10.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:10.185+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:10.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:10.185+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:10.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:10.185+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:10.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:10.186+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:10.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:10.376+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:10.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:10.376+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:10.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:10.376+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:10.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:10.376+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:11.212 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:10.860+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:11.212 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:10.860+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:11.212 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:10.860+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:11.212 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:10.861+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:11.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:11.418+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:11.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:11.418+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:11.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:11.418+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:11.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:11.419+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:11.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:11.208+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:11.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:11.208+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:11.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:11.208+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:11.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:11.209+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:12.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:11.835+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:12.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:11.835+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:12.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:11.835+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:12.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:11.836+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:12.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:12.247+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:12.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:12.247+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:12.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:12.247+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:12.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:12.248+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:12.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:12.411+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:12.532 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:12.411+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:12.532 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:12.411+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:12.532 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:12.412+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:13.230 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:12.825+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:13.231 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:12.825+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:13.231 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:12.825+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:13.231 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:12.826+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:13.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:13.441+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:13.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:13.441+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:13.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:13.441+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:13.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:13.441+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:13.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:13.226+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:13.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:13.226+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:13.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:13.226+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:13.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:13.227+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:14.243 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:13.820+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:14.243 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:13.820+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:14.243 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:13.820+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:14.243 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:13.821+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:14.507 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:10:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:10:14.381+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 4016 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:10:14.508 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:14.239+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:14.508 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:14.239+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:14.508 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:14.239+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:14.508 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:14.240+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:14.508 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:14.425+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:14.508 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:14.425+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:14.508 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:14.425+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:14.509 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:14.426+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:15.268 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:14.819+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:15.269 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:14.819+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:15.269 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:14.819+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:15.269 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:14.820+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:15.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:15.265+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:15.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:15.265+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:15.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:15.265+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:15.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:15.265+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:15.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:15.409+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:15.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:15.409+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:15.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:15.409+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:15.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:15.409+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:16.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:15.865+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:16.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:15.865+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:16.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:15.865+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:16.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:15.866+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:16.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:16.304+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:16.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:16.304+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:16.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:16.304+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:16.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:16.305+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:16.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:16.403+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:16.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:16.403+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:16.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:16.403+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:16.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:16.403+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:17.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:16.830+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:17.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:16.830+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:17.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:16.830+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:17.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:16.831+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:17.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:17.312+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:17.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:17.312+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:17.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:17.312+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:17.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:17.312+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:17.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:17.394+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:17.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:17.394+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:17.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:17.394+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:17.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:17.394+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:18.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:17.787+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:18.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:17.787+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:18.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:17.787+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:18.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:17.787+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:18.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:18.290+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:18.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:18.290+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:18.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:18.290+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:18.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:18.290+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:18.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:18.400+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:18.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:18.400+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:18.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:18.400+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:18.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:18.401+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:19.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:18.788+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:19.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:18.788+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:19.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:18.788+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:19.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:18.789+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:19.771 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:19.277+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:19.771 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:19.277+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:19.771 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:19.277+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:19.771 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:19.277+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:19.772 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:19.351+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:19.772 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:19.351+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:19.772 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:19.351+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:19.772 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:19.351+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:19.772 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:10:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:10:19.382+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 4110 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:10:20.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:19.767+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:20.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:19.767+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:20.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:19.767+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:20.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:19.767+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:20.400 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:20.292+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:20.400 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:20.292+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:20.400 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:20.292+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:20.400 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:20.293+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:20.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:20.396+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:20.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:20.396+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:20.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:20.396+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:20.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:20.397+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:21.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:20.806+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:21.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:20.806+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:21.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:20.806+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:21.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:20.807+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:21.654 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:21.423+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:21.654 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:21.423+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:21.654 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:21.423+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:21.654 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:21.424+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:21.655 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:21.335+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:21.655 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:21.335+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:21.655 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:21.335+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:21.655 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:21.336+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:22.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:21.773+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:22.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:21.773+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:22.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:21.773+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:22.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:21.774+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:22.747 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:22.464+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:22.747 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:22.464+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:22.747 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:22.464+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:22.747 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:22.465+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:22.747 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:22.353+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:22.748 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:22.353+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:22.748 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:22.353+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:22.748 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:22.354+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:23.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:22.743+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:23.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:22.743+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:23.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:22.743+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:23.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:22.744+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:23.459 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:23.351+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:23.460 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:23.351+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:23.460 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:23.351+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:23.460 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:23.352+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:23.757 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:23.455+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:23.757 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:23.455+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:23.757 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:23.455+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:23.757 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:23.456+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:24.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:23.753+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:24.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:23.753+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:24.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:23.753+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:24.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:23.754+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:24.712 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:10:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:10:24.382+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 4206 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:10:24.712 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:24.480+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:24.712 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:24.480+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:24.712 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:24.480+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:24.712 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:24.481+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:24.713 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:24.352+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:24.713 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:24.352+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:24.713 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:24.352+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:24.713 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:24.353+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:25.029 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:24.707+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:25.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:24.707+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:25.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:24.707+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:25.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:24.708+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:25.762 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:25.311+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:25.762 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:25.311+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:25.763 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:25.311+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:25.763 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:25.311+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:25.763 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:25.493+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:25.763 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:25.493+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:25.763 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:25.493+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:25.763 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:25.494+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:26.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:25.758+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:26.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:25.758+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:26.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:25.758+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:26.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:25.759+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:26.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:26.307+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:26.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:26.307+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:26.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:26.307+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:26.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:26.308+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:27.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:26.539+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:27.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:26.539+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:27.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:26.539+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:27.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:26.540+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:27.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:26.741+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:27.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:26.741+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:27.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:26.741+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:27.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:26.742+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:27.519 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:27.261+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:27.519 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:27.261+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:27.519 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:27.261+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:27.520 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:27.262+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:27.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:27.759+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:27.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:27.759+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:27.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:27.759+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:27.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:27.760+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:27.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:27.515+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:27.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:27.515+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:27.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:27.515+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:27.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:27.516+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:28.545 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:28.289+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:28.546 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:28.289+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:28.546 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:28.289+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:28.546 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:28.290+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:29.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:28.759+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:29.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:28.759+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:29.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:28.759+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:29.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:28.760+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:29.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:28.541+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:29.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:28.541+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:29.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:28.541+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:29.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:28.542+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:29.543 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:10:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:10:29.383+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 4306 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:10:29.543 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:29.278+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:29.543 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:29.278+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:29.543 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:29.278+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:29.544 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:29.278+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:30.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:29.739+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:30.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:29.739+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:30.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:29.739+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:30.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:29.740+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:30.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:29.539+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:30.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:29.539+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:30.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:29.539+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:30.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:29.540+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:30.546 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:30.278+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:30.547 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:30.278+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:30.547 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:30.278+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:30.547 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:30.279+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:31.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:30.767+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:31.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:30.767+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:31.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:30.767+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:31.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:30.768+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:31.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:30.542+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:31.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:30.543+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:31.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:30.543+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:31.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:30.544+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:31.546 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:31.286+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:31.546 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:31.286+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:31.546 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:31.286+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:31.546 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:31.286+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:31.821 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:31.542+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:31.821 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:31.542+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:31.821 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:31.542+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:31.822 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:31.543+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:32.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:31.817+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:32.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:31.817+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:32.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:31.817+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:32.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:31.818+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:32.573 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:32.288+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:32.573 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:32.288+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:32.573 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:32.288+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:32.573 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:32.289+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:32.836 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:32.569+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:32.836 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:32.569+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:32.836 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:32.569+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:32.836 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:32.570+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:33.245 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:32.832+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:33.245 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:32.832+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:33.245 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:32.832+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:33.245 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:32.833+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:33.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:33.241+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:33.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:33.241+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:33.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:33.241+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:33.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:33.242+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:33.799 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:33.612+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:33.800 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:33.612+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:33.800 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:33.612+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:33.800 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:33.613+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:34.196 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:33.795+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:34.197 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:33.795+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:34.197 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:33.795+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:34.197 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:33.796+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:34.511 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:10:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:10:34.384+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 4402 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:10:34.512 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:34.192+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:34.512 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:34.192+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:34.512 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:34.192+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:34.512 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:34.193+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:34.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:34.599+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:34.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:34.599+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:34.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:34.599+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:34.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:34.600+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:35.189 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:34.831+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:35.189 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:34.831+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:35.189 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:34.831+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:35.189 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:34.832+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:35.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:35.184+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:35.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:35.184+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:35.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:35.184+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:35.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:35.185+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:35.883 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:35.879+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:35.929 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:35.629+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:35.929 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:35.629+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:35.929 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:35.629+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:35.929 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:35.630+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:36.208 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:35.879+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:36.209 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:35.879+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:36.209 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:35.880+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:36.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:36.204+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:36.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:36.204+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:36.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:36.204+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:36.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:36.205+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:36.933 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:36.642+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:36.933 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:36.642+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:36.933 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:36.642+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:36.933 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:36.642+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:37.194 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:36.929+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:37.194 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:36.929+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:37.194 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:36.929+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:37.194 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:36.930+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:37.529 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:37.190+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:37.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:37.190+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:37.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:37.190+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:37.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:37.190+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:37.979 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:37.634+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:37.980 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:37.634+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:37.980 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:37.634+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:37.980 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:37.635+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:38.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:37.975+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:38.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:37.975+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:38.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:37.975+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:38.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:37.976+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:38.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:38.196+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:38.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:38.196+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:38.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:38.196+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:38.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:38.197+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:39.012 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:38.640+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:39.012 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:38.640+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:39.012 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:38.640+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:39.012 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:38.641+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:39.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:39.008+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:39.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:39.008+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:39.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:39.008+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:39.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:39.009+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:39.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:39.219+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:39.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:39.219+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:39.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:39.219+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:39.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:39.220+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:39.779 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:10:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:10:39.385+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 4495 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:10:39.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:39.624+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:39.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:39.624+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:39.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:39.624+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:39.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:39.625+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:40.274 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:39.974+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:40.274 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:39.974+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:40.274 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:39.974+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:40.274 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:39.975+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:40.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:40.270+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:40.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:40.270+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:40.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:40.270+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:40.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:40.270+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:40.981 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:40.609+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:40.981 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:40.609+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:40.981 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:40.609+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:40.981 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:40.610+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:41.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:40.977+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:41.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:40.977+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:41.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:40.977+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:41.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:40.978+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:41.655 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:41.305+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:41.655 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:41.305+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:41.655 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:41.305+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:41.656 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:41.306+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:42.024 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:41.657+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:42.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:41.657+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:42.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:41.657+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:42.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:41.658+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:42.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:42.020+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:42.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:42.020+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:42.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:42.020+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:42.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:42.021+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:42.700 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:42.356+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:42.700 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:42.356+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:42.700 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:42.356+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:42.700 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:42.356+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:42.979 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:42.696+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:42.979 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:42.696+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:42.979 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:42.696+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:42.979 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:42.697+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:43.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:42.975+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:43.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:42.975+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:43.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:42.975+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:43.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:42.976+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:43.744 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:43.317+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:43.744 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:43.317+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:43.744 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:43.317+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:43.744 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:43.318+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:44.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:43.739+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:44.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:43.739+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:44.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:43.739+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:44.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:43.740+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:44.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:44.023+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:44.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:44.023+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:44.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:44.023+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:44.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:44.024+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:44.646 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:10:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:10:44.386+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 4589 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:10:44.646 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:44.334+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:44.646 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:44.334+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:44.646 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:44.334+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:44.646 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:44.335+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:45.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:44.698+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:45.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:44.698+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:45.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:44.698+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:45.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:44.699+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:45.358 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:45.072+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:45.358 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:45.072+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:45.358 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:45.072+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:45.358 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:45.073+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:45.740 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:45.353+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:45.740 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:45.353+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:45.740 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:45.353+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:45.740 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:45.354+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:46.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:45.736+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:46.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:45.736+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:46.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:45.736+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:46.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:45.737+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:46.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:46.023+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:46.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:46.023+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:46.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:46.023+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:46.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:46.024+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:46.749 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:46.369+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:46.749 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:46.369+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:46.749 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:46.369+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:46.749 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:46.370+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:47.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:46.744+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:47.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:46.744+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:47.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:46.744+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:47.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:46.746+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:47.385 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:47.041+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:47.385 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:47.041+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:47.385 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:47.041+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:47.386 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:47.042+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:47.755 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:47.381+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:47.756 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:47.381+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:47.756 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:47.381+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:47.756 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:47.382+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:48.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:47.751+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:48.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:47.751+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:48.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:47.751+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:48.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:47.752+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:48.399 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:48.055+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:48.399 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:48.055+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:48.400 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:48.055+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:48.400 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:48.056+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:48.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:48.395+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:48.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:48.395+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:48.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:48.395+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:48.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:48.396+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:49.097 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:48.799+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:49.097 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:48.799+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:49.098 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:48.799+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:49.098 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:48.800+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:49.388 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:49.093+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:49.388 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:49.093+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:49.388 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:49.093+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:49.388 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:49.094+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:49.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:10:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:10:49.386+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 4700 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:10:49.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:49.423+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:49.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:49.423+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:49.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:49.423+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:49.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:49.424+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:50.109 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:49.820+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:50.109 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:49.820+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:50.109 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:49.820+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:50.109 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:49.821+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:50.392 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:50.105+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:50.392 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:50.105+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:50.392 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:50.105+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:50.392 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:50.106+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:50.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:50.388+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:50.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:50.388+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:50.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:50.388+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:50.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:50.388+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:51.135 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:50.797+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:51.136 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:50.797+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:51.136 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:50.797+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:51.136 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:50.798+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:51.429 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:51.131+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:51.429 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:51.131+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:51.429 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:51.131+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:51.429 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:51.132+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:51.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:51.424+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:51.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:51.425+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:51.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:51.425+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:51.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:51.425+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:52.175 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:51.827+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:52.175 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:51.827+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:52.175 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:51.827+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:52.175 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:51.828+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:52.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:52.171+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:52.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:52.171+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:52.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:52.171+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:52.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:52.172+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:52.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:52.380+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:52.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:52.380+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:52.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:52.380+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:52.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:52.380+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:53.207 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:52.853+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:53.208 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:52.853+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:53.208 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:52.853+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:53.208 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:52.854+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:53.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:53.203+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:53.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:53.203+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:53.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:53.203+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:53.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:53.204+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:53.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:53.404+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:53.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:53.404+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:53.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:53.404+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:53.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:53.405+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:54.160 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:53.805+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:54.160 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:53.805+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:54.160 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:53.805+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:54.160 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:53.806+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:54.413 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:10:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:10:54.387+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 4793 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:10:54.413 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:54.392+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:54.413 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:54.392+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:54.413 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:54.392+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:54.414 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:54.394+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:54.414 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:54.155+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:54.414 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:54.155+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:54.414 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:54.155+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:54.414 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:54.156+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:55.200 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:54.824+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:55.200 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:54.824+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:55.200 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:54.824+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:55.200 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:54.825+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:55.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:55.388+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:55.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:55.388+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:55.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:55.388+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:55.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:55.389+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:55.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:55.196+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:55.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:55.196+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:55.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:55.196+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:55.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:55.197+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:56.196 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:55.798+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:56.196 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:55.798+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:56.196 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:55.798+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:56.196 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:55.798+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:56.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:56.357+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:56.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:56.357+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:56.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:56.357+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:56.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:56.358+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:56.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:56.192+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:56.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:56.192+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:56.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:56.192+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:56.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:56.192+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:57.188 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:56.832+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:57.188 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:56.832+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:57.188 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:56.832+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:57.188 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:56.832+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:57.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:57.345+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:57.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:57.345+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:57.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:57.345+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:57.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:57.346+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:57.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:57.184+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:57.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:57.184+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:57.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:57.184+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:57.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:57.184+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:58.226 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:57.785+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:58.226 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:57.785+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:58.226 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:57.785+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:58.227 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:57.786+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:58.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:58.222+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:58.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:58.222+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:58.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:58.222+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:58.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:58.223+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:58.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:58.357+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:58.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:58.357+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:58.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:58.357+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:58.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:58.358+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:59.254 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:58.816+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:59.254 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:58.816+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:59.254 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:58.816+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:10:59.254 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:58.817+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:59.516 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:59.250+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:59.516 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:59.250+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:59.516 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:59.250+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:10:59.516 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:10:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:10:59.250+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:59.517 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:59.348+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:59.517 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:59.348+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:59.517 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:59.349+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:10:59.517 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:10:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:10:59.349+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:10:59.517 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:10:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:10:59.388+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 4886 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:11:00.248 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:59.854+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:00.248 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:59.854+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:00.248 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:59.854+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:00.248 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:10:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:10:59.855+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:00.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:00.244+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:00.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:00.244+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:00.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:00.244+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:00.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:00.245+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:00.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:00.352+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:00.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:00.352+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:00.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:00.352+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:00.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:00.353+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:01.211 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:00.838+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:01.212 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:00.838+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:01.212 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:00.838+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:01.212 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:00.839+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:01.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:01.207+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:01.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:01.207+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:01.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:01.207+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:01.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:01.208+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:01.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:01.323+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:01.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:01.323+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:01.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:01.323+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:01.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:01.323+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:02.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:01.803+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:02.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:01.803+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:02.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:01.803+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:02.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:01.804+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:02.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:02.304+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:02.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:02.304+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:02.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:02.304+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:02.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:02.305+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:02.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:02.200+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:02.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:02.200+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:02.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:02.200+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:02.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:02.201+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:03.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:02.755+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:03.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:02.755+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:03.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:02.755+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:03.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:02.755+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:03.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:03.181+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:03.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:03.182+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:03.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:03.182+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:03.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:03.183+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:03.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:03.314+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:03.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:03.314+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:03.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:03.314+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:03.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:03.315+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:04.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:03.773+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:04.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:03.773+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:04.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:03.773+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:04.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:03.773+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:04.515 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:11:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:11:04.389+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 4985 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:11:04.515 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:04.198+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:04.515 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:04.198+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:04.515 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:04.198+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:04.515 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:04.199+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:04.515 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:04.296+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:04.516 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:04.296+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:04.516 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:04.296+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:04.516 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:04.297+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:05.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:04.761+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:05.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:04.761+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:05.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:04.761+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:05.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:04.762+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:05.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:05.240+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:05.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:05.240+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:05.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:05.240+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:05.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:05.241+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:05.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:05.289+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:05.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:05.289+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:05.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:05.289+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:05.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:05.290+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:06.150 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:05.797+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:06.150 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:05.797+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:06.150 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:05.797+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:06.150 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:05.797+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:06.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:06.328+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:06.598 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:06.328+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:06.598 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:06.328+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:06.599 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:06.328+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:06.599 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:06.269+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:06.599 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:06.269+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:06.599 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:06.269+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:06.599 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:06.270+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:07.225 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:06.820+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:07.225 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:06.820+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:07.225 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:06.820+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:07.225 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:06.821+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:07.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:07.221+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:07.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:07.221+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:07.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:07.221+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:07.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:07.222+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:07.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:07.339+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:07.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:07.339+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:07.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:07.339+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:07.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:07.340+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:08.252 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:07.811+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:08.252 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:07.811+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:08.252 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:07.811+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:08.252 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:07.812+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:08.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:08.248+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:08.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:08.248+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:08.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:08.248+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:08.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:08.249+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:08.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:08.304+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:08.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:08.304+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:08.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:08.304+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:08.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:08.305+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:09.223 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:08.777+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:09.223 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:08.777+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:09.223 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:08.777+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:09.223 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:08.778+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:09.518 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:11:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:11:09.390+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 5093 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:11:09.518 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:09.260+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:09.518 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:09.260+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:09.519 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:09.260+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:09.519 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:09.261+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:09.519 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:09.257+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:09.519 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:09.257+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:09.519 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:09.257+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:09.519 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:09.258+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:09.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:09.735+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:09.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:09.735+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:09.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:09.735+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:09.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:09.736+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:10.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:10.241+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:10.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:10.241+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:10.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:10.241+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:10.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:10.242+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:10.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:10.283+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:10.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:10.283+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:10.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:10.283+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:10.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:10.284+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:11.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:10.715+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:11.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:10.715+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:11.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:10.715+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:11.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:10.716+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:11.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:11.250+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:11.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:11.250+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:11.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:11.250+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:11.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:11.251+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:11.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:11.256+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:11.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:11.256+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:11.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:11.256+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:11.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:11.257+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:12.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:11.719+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:12.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:11.719+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:12.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:11.719+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:12.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:11.720+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:12.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:12.207+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:12.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:12.207+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:12.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:12.207+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:12.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:12.208+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:12.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:12.210+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:12.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:12.210+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:12.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:12.210+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:12.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:12.211+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:13.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:12.689+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:13.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:12.689+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:13.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:12.689+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:13.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:12.690+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:13.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:13.162+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:13.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:13.162+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:13.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:13.162+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:13.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:13.163+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:13.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:13.164+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:13.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:13.164+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:13.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:13.164+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:13.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:13.164+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:14.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:13.683+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:14.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:13.683+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:14.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:13.683+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:14.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:13.684+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:14.415 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:14.208+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:14.415 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:14.208+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:14.416 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:14.208+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:14.416 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:14.209+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:14.416 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:14.156+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:14.416 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:14.156+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:14.416 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:14.156+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:14.416 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:14.157+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:14.417 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:11:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:11:14.391+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 5201 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:11:15.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:14.726+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:15.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:14.726+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:15.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:14.726+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:15.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:14.727+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:15.367 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:15.111+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:15.367 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:15.111+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:15.367 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:15.111+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:15.367 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:15.112+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:15.367 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:15.170+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:15.368 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:15.170+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:15.368 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:15.170+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:15.368 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:15.171+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:16.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:15.773+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:16.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:15.773+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:16.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:15.773+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:16.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:15.774+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:16.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:16.148+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:16.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:16.148+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:16.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:16.148+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:16.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:16.148+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:16.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:16.126+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:16.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:16.126+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:16.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:16.126+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:16.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:16.127+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:17.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:16.728+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:17.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:16.728+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:17.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:16.728+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:17.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:16.728+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:17.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:17.182+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:17.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:17.182+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:17.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:17.182+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:17.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:17.183+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:17.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:17.096+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:17.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:17.096+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:17.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:17.096+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:17.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:17.096+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:18.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:17.762+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:18.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:17.762+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:18.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:17.762+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:18.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:17.763+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:18.439 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:18.137+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:18.439 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:18.137+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:18.439 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:18.137+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:18.439 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:18.138+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:18.439 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:18.115+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:18.439 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:18.115+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:18.439 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:18.115+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:18.439 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:18.116+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:19.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:18.739+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:19.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:18.739+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:19.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:18.739+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:19.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:18.739+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:19.388 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:19.158+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:19.388 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:19.158+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:19.388 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:19.158+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:19.388 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:19.159+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:19.388 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:19.095+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:19.389 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:19.095+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:19.389 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:19.095+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:19.389 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:19.096+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:19.697 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:11:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:11:19.392+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 5294 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:11:20.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:19.693+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:20.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:19.693+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:20.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:19.693+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:20.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:19.694+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:20.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:20.136+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:20.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:20.136+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:20.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:20.136+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:20.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:20.137+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:20.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:20.126+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:20.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:20.126+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:20.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:20.126+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:20.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:20.127+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:21.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:20.725+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:21.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:20.725+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:21.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:20.725+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:21.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:20.726+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:21.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:21.168+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:21.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:21.168+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:21.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:21.168+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:21.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:21.169+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:21.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:21.092+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:21.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:21.092+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:21.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:21.092+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:21.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:21.092+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:22.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:21.738+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:22.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:21.738+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:22.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:21.738+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:22.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:21.738+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:22.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:22.199+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:22.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:22.199+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:22.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:22.199+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:22.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:22.200+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:22.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:22.108+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:22.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:22.108+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:22.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:22.108+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:22.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:22.109+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:23.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:22.741+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:23.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:22.741+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:23.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:22.741+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:23.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:22.742+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:23.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:23.177+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:23.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:23.177+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:23.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:23.177+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:23.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:23.178+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:23.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:23.140+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:23.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:23.140+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:23.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:23.140+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:23.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:23.140+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:24.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:23.752+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:24.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:23.752+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:24.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:23.752+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:24.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:23.752+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:24.397 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:24.126+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:24.397 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:24.126+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:24.397 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:24.126+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:24.397 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:24.127+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:24.397 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:24.134+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:24.397 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:24.134+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:24.398 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:24.134+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:24.398 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:24.135+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:24.715 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:11:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:11:24.393+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 5387 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:11:25.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:24.711+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:25.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:24.711+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:25.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:24.711+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:25.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:24.712+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:25.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:25.176+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:25.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:25.176+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:25.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:25.176+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:25.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:25.177+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:25.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:25.154+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:25.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:25.154+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:25.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:25.154+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:25.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:25.155+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:26.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:25.718+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:26.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:25.718+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:26.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:25.718+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:26.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:25.719+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:26.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:26.189+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:26.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:26.189+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:26.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:26.189+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:26.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:26.190+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:26.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:26.164+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:26.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:26.164+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:26.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:26.164+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:26.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:26.165+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:27.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:26.733+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:27.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:26.733+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:27.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:26.733+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:27.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:26.734+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:27.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:27.232+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:27.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:27.232+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:27.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:27.232+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:27.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:27.233+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:27.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:27.204+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:27.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:27.204+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:27.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:27.204+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:27.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:27.204+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:28.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:27.689+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:28.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:27.689+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:28.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:27.689+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:28.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:27.690+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:28.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:28.244+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:28.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:28.244+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:28.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:28.244+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:28.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:28.244+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:28.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:28.190+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:28.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:28.190+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:28.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:28.190+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:28.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:28.191+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:29.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:28.644+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:29.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:28.644+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:29.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:28.644+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:29.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:28.644+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:29.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:11:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:11:29.394+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 5480 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:11:29.522 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:29.189+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:29.522 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:29.189+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:29.522 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:29.189+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:29.522 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:29.189+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:29.523 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:29.220+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:29.523 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:29.220+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:29.523 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:29.220+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:29.523 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:29.221+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:29.779 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:29.614+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:29.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:29.614+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:29.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:29.614+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:29.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:29.614+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:30.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:30.149+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:30.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:30.149+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:30.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:30.149+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:30.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:30.150+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:30.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:30.255+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:30.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:30.255+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:30.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:30.255+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:30.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:30.256+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:31.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:30.582+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:31.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:30.582+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:31.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:30.582+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:31.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:30.583+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:31.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:31.281+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:31.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:31.281+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:31.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:31.281+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:31.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:31.282+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:31.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:31.172+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:31.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:31.172+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:31.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:31.172+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:31.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:31.172+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:32.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:31.540+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:32.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:31.540+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:32.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:31.540+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:32.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:31.541+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:32.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:32.216+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:32.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:32.216+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:32.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:32.216+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:32.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:32.217+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:32.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:32.328+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:32.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:32.328+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:32.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:32.328+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:32.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:32.328+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:33.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:32.550+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:33.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:32.550+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:33.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:32.550+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:33.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:32.551+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:33.523 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:33.362+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:33.524 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:33.362+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:33.524 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:33.362+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:33.524 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:33.363+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:33.524 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:33.254+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:33.524 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:33.254+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:33.524 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:33.254+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:33.524 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:33.254+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:33.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:33.519+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:33.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:33.519+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:33.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:33.519+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:33.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:33.520+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:34.522 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:11:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:11:34.394+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 5572 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:11:34.523 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:34.319+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:34.523 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:34.319+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:34.523 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:34.319+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:34.523 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:34.320+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:34.523 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:34.237+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:34.523 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:34.237+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:34.524 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:34.237+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:34.524 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:34.238+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:34.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:34.529+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:34.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:34.529+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:34.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:34.529+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:34.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:34.530+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:35.524 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:35.315+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:35.524 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:35.315+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:35.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:35.315+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:35.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:35.316+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:35.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:35.254+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:35.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:35.254+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:35.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:35.254+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:35.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:35.255+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:35.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:35.520+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:35.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:35.520+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:35.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:35.520+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:35.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:35.521+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:36.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:36.290+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:36.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:36.290+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:36.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:36.290+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:36.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:36.291+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:36.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:36.359+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:36.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:36.359+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:36.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:36.359+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:36.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:36.359+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:36.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:36.504+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:36.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:36.504+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:36.782 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:36.504+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:36.782 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:36.505+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:37.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:37.247+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:37.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:37.247+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:37.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:37.247+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:37.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:37.247+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:37.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:37.403+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:37.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:37.403+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:37.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:37.403+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:37.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:37.404+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:38.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:37.527+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:38.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:37.527+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:38.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:37.527+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:38.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:37.528+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:38.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:38.370+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:38.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:38.370+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:38.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:38.370+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:38.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:38.371+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:38.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:38.201+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:38.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:38.201+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:38.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:38.201+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:38.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:38.202+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:39.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:38.567+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:39.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:38.567+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:39.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:38.567+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:39.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:38.568+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:39.351 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:39.228+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:39.351 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:39.228+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:39.351 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:39.228+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:39.351 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:39.228+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:39.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:11:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:11:39.395+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 5664 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:11:39.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:39.347+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:39.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:39.347+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:39.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:39.347+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:39.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:39.347+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:39.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:39.544+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:39.782 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:39.544+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:39.782 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:39.544+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:39.782 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:39.545+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:40.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:40.275+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:40.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:40.275+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:40.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:40.275+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:40.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:40.275+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:40.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:40.338+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:40.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:40.338+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:40.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:40.338+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:40.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:40.339+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:40.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:40.517+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:40.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:40.517+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:40.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:40.517+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:40.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:40.518+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:41.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:41.383+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:41.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:41.383+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:41.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:41.383+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:41.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:41.384+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:41.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:41.494+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:41.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:41.494+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:41.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:41.494+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:41.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:41.494+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:41.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:41.252+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:41.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:41.252+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:41.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:41.252+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:41.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:41.253+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:42.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:42.423+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:42.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:42.423+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:42.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:42.423+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:42.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:42.424+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:42.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:42.479+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:42.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:42.479+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:42.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:42.479+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:42.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:42.480+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:42.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:42.247+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:42.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:42.247+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:42.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:42.247+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:42.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:42.248+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:43.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:43.267+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:43.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:43.267+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:43.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:43.267+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:43.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:43.268+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:43.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:43.444+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:43.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:43.444+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:43.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:43.444+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:43.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:43.445+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:43.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:43.477+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:43.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:43.477+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:43.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:43.477+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:43.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:43.478+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:44.525 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:11:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:11:44.396+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 5757 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:11:44.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:44.441+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:44.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:44.441+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:44.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:44.441+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:44.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:44.442+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:44.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:44.247+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:44.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:44.247+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:44.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:44.247+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:44.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:44.248+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:44.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:44.527+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:44.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:44.527+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:44.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:44.527+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:44.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:44.528+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:45.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:45.451+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:45.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:45.451+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:45.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:45.451+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:45.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:45.452+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:45.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:45.208+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:45.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:45.208+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:45.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:45.208+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:45.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:45.208+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:46.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:45.575+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:46.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:45.575+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:46.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:45.575+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:46.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:45.576+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:46.494 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:46.250+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:46.494 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:46.250+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:46.494 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:46.250+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:46.494 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:46.251+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:46.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:46.490+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:46.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:46.490+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:46.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:46.490+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:46.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:46.491+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:46.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:46.615+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:46.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:46.615+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:46.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:46.615+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:46.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:46.615+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:47.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:47.502+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:47.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:47.502+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:47.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:47.502+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:47.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:47.504+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:47.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:47.268+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:47.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:47.268+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:47.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:47.268+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:47.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:47.269+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:48.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:47.618+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:48.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:47.618+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:48.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:47.618+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:48.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:47.618+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:48.596 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:48.489+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:48.596 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:48.489+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:48.596 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:48.489+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:48.596 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:48.490+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:48.597 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:48.302+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:48.597 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:48.302+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:48.597 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:48.302+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:48.597 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:48.303+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:49.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:48.592+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:49.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:48.592+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:49.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:48.592+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:49.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:48.593+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:49.612 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:11:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:11:49.397+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 5850 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:11:49.612 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:49.449+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:49.612 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:49.449+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:49.613 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:49.449+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:49.613 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:49.450+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:49.613 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:49.345+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:49.613 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:49.345+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:49.613 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:49.345+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:49.613 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:49.346+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:50.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:49.608+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:50.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:49.608+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:50.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:49.608+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:50.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:49.609+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:50.648 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:50.459+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:50.649 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:50.459+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:50.649 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:50.459+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:50.649 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:50.459+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:50.649 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:50.391+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:50.649 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:50.391+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:50.649 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:50.391+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:50.649 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:50.392+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:51.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:50.644+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:51.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:50.644+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:51.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:50.644+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:51.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:50.645+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:51.660 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:51.480+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:51.660 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:51.480+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:51.660 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:51.480+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:51.660 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:51.481+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:51.660 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:51.389+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:51.661 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:51.389+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:51.661 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:51.389+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:51.661 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:51.390+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:51.661 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:51.612+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:51.661 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:51.612+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:51.661 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:51.612+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:51.661 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:51.613+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:52.503 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:52.378+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:52.503 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:52.378+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:52.503 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:52.378+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:52.503 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:52.379+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:52.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:52.499+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:52.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:52.499+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:52.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:52.499+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:52.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:52.500+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:52.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:52.615+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:52.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:52.615+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:52.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:52.615+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:52.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:52.616+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:53.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:53.380+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:53.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:53.380+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:53.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:53.380+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:53.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:53.381+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:53.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:53.584+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:53.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:53.584+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:53.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:53.584+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:53.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:53.585+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:53.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:53.499+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:53.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:53.499+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:53.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:53.499+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:53.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:53.501+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:54.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:11:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:11:54.398+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 5942 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:11:54.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:54.375+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:54.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:54.375+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:54.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:54.375+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:54.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:54.376+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:54.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:54.566+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:54.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:54.566+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:54.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:54.566+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:54.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:54.566+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:54.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:54.516+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:54.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:54.516+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:54.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:54.516+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:54.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:54.516+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:55.562 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:55.478+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:55.562 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:55.479+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:55.562 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:55.479+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:55.562 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:55.479+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:55.562 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:55.342+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:55.563 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:55.342+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:55.563 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:55.342+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:55.563 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:55.343+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:56.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:55.558+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:56.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:55.558+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:56.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:55.558+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:56.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:55.559+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:56.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:56.347+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:56.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:56.347+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:56.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:56.347+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:56.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:56.348+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:56.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:56.552+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:56.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:56.552+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:56.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:56.552+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:56.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:56.553+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:56.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:56.492+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:56.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:56.492+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:56.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:56.492+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:56.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:56.493+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:57.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:57.471+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:57.830 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:57.471+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:57.830 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:57.471+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:57.830 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:57.471+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:57.831 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:57.302+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:57.831 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:57.302+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:57.831 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:57.302+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:57.831 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:57.303+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:57.831 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:57.531+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:57.831 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:57.531+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:57.831 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:57.531+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:57.832 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:57.532+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:58.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:58.501+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:58.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:58.501+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:58.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:58.501+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:58.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:58.502+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:58.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:58.516+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:58.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:58.516+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:58.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:58.516+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:58.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:58.517+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:58.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:58.288+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:58.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:58.288+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:58.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:58.288+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:58.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:58.289+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:59.529 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:59.274+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:59.529 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:59.274+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:59.529 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:59.274+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:11:59.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:11:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:11:59.274+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:59.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:11:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:11:59.400+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 6034 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:11:59.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:59.505+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:59.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:59.505+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:59.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:59.505+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:11:59.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:11:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:11:59.506+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:11:59.779 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:59.560+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:59.779 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:59.560+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:59.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:59.560+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:11:59.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:11:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:11:59.562+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:00.586 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:00.479+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:00.586 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:00.479+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:00.586 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:00.479+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:00.586 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:00.479+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:00.587 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:00.282+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:00.587 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:00.282+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:00.587 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:00.282+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:00.587 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:00.282+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:01.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:00.582+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:01.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:00.582+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:01.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:00.582+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:01.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:00.583+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:01.565 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:01.514+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:01.565 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:01.514+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:01.565 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:01.514+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:01.565 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:01.516+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:01.566 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:01.310+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:01.566 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:01.310+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:01.566 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:01.310+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:01.566 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:01.311+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:02.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:01.561+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:02.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:01.561+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:02.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:01.561+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:02.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:01.562+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:02.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:02.525+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:02.529 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:02.275+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:02.529 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:02.275+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:02.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:02.275+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:02.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:02.276+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:02.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:02.492+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:02.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:02.492+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:02.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:02.492+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:02.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:02.493+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:02.779 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:02.525+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:02.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:02.525+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:02.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:02.526+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:03.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:03.479+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:03.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:03.479+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:03.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:03.479+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:03.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:03.481+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:03.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:03.261+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:03.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:03.261+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:03.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:03.261+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:03.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:03.262+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:04.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:03.532+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:04.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:03.532+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:04.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:03.532+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:04.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:03.533+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:04.514 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:12:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:12:04.400+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 6126 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:12:04.514 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:04.229+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:04.514 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:04.229+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:04.514 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:04.229+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:04.515 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:04.230+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:04.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:04.540+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:04.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:04.540+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:04.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:04.540+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:04.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:04.541+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:04.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:04.510+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:04.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:04.510+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:04.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:04.510+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:04.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:04.511+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:05.474 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:05.210+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:05.474 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:05.210+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:05.474 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:05.210+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:05.475 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:05.211+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:05.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:05.588+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:05.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:05.588+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:05.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:05.588+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:05.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:05.589+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:05.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:05.470+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:05.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:05.470+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:05.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:05.470+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:05.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:05.471+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:06.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:06.428+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:06.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:06.428+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:06.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:06.428+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:06.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:06.428+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:06.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:06.238+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:06.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:06.238+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:06.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:06.238+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:06.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:06.238+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:07.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:06.613+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:07.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:06.613+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:07.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:06.613+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:07.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:06.614+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:07.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:07.274+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:07.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:07.274+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:07.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:07.274+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:07.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:07.275+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:07.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:07.435+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:07.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:07.435+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:07.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:07.435+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:07.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:07.436+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:08.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:07.656+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:08.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:07.656+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:08.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:07.656+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:08.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:07.657+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:08.654 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:08.286+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:08.654 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:08.286+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:08.655 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:08.286+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:08.655 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:08.286+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:08.655 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:08.420+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:08.655 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:08.420+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:08.655 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:08.420+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:08.655 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:08.421+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:09.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:08.650+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:09.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:08.650+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:09.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:08.650+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:09.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:08.651+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:09.698 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:12:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:12:09.401+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 6220 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:12:09.698 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:09.456+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:09.698 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:09.456+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:09.698 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:09.456+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:09.698 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:09.457+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:09.698 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:09.307+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:09.698 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:09.307+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:09.699 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:09.307+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:09.699 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:09.307+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:10.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:09.694+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:10.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:09.694+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:10.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:09.694+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:10.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:09.695+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:10.663 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:10.312+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:10.664 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:10.312+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:10.664 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:10.312+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:10.664 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:10.313+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:10.664 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:10.429+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:10.664 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:10.429+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:10.664 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:10.429+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:10.664 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:10.430+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:11.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:10.689+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:11.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:10.689+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:11.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:10.690+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:11.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:10.690+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:11.661 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:11.292+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:11.661 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:11.292+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:11.661 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:11.292+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:11.662 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:11.293+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:11.662 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:11.424+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:11.662 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:11.424+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:11.662 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:11.424+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:11.662 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:11.425+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:12.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:11.658+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:12.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:11.658+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:12.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:11.658+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:12.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:11.658+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:12.620 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:12.334+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:12.621 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:12.334+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:12.621 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:12.335+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:12.621 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:12.335+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:12.621 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:12.392+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:12.621 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:12.392+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:12.621 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:12.392+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:12.621 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:12.393+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:13.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:12.616+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:13.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:12.616+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:13.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:12.616+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:13.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:12.617+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:13.734 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:13.579+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:13.734 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:13.579+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:13.734 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:13.579+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:13.734 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:13.580+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:13.735 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:13.380+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:13.735 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:13.380+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:13.735 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:13.380+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:13.735 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:13.381+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:13.735 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:13.406+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:13.735 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:13.406+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:13.735 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:13.406+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:13.735 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:13.407+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:14.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:14.564+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:14.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:14.564+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:14.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:14.564+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:14.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:14.565+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:14.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:14.376+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:14.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:14.376+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:14.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:14.376+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:14.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:14.377+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:14.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:14.438+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:14.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:14.438+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:14.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:14.438+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:14.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:14.439+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:14.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:12:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:12:14.402+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 6313 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:12:15.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:15.525+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:15.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:15.525+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:15.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:15.525+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:15.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:15.526+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:15.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:15.343+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:15.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:15.343+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:15.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:15.343+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:15.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:15.344+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:15.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:15.418+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:15.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:15.418+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:15.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:15.418+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:15.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:15.419+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:16.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:16.500+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:16.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:16.500+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:16.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:16.500+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:16.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:16.501+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:16.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:16.364+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:16.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:16.364+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:16.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:16.364+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:16.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:16.365+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:16.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:16.438+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:16.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:16.438+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:16.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:16.438+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:16.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:16.439+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:17.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:17.520+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:17.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:17.520+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:17.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:17.520+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:17.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:17.521+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:17.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:17.339+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:17.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:17.339+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:17.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:17.339+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:17.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:17.339+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:17.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:17.439+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:17.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:17.439+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:17.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:17.439+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:17.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:17.439+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:18.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:18.477+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:18.782 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:18.477+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:18.782 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:18.477+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:18.782 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:18.478+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:18.782 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:18.347+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:18.782 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:18.347+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:18.782 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:18.347+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:18.782 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:18.348+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:18.783 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:18.485+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:18.783 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:18.485+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:18.783 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:18.485+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:18.783 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:18.486+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:19.779 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:12:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:12:19.403+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 6406 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:12:19.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:19.354+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:19.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:19.354+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:19.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:19.354+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:19.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:19.354+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:19.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:19.449+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:19.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:19.449+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:19.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:19.449+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:19.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:19.450+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:19.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:19.458+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:19.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:19.458+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:19.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:19.458+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:19.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:19.459+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:20.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:20.380+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:20.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:20.380+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:20.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:20.380+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:20.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:20.380+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:20.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:20.497+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:20.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:20.497+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:20.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:20.497+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:20.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:20.497+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:20.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:20.503+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:20.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:20.503+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:20.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:20.503+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:20.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:20.504+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:21.662 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:21.338+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:21.662 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:21.338+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:21.662 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:21.338+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:21.662 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:21.339+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:21.663 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:21.476+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:21.663 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:21.476+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:21.663 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:21.476+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:21.663 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:21.476+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:21.663 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:21.551+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:21.663 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:21.551+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:21.663 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:21.551+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:21.663 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:21.552+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:22.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:22.459+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:22.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:22.459+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:22.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:22.459+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:22.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:22.460+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:22.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:22.547+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:22.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:22.547+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:22.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:22.547+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:22.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:22.548+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:22.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:22.339+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:22.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:22.339+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:22.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:22.339+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:22.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:22.340+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:23.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:23.537+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:23.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:23.537+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:23.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:23.537+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:23.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:23.538+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:23.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:23.474+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:23.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:23.474+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:23.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:23.474+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:23.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:23.475+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:23.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:23.316+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:23.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:23.316+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:23.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:23.316+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:23.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:23.317+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:24.531 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:12:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:12:24.404+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 6498 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:12:24.532 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:24.527+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:24.532 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:24.527+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:24.532 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:24.527+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:24.532 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:24.528+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:24.532 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:24.471+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:24.532 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:24.471+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:24.533 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:24.471+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:24.533 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:24.472+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:24.533 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:24.277+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:24.533 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:24.277+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:24.533 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:24.277+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:24.533 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:24.278+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:25.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:25.271+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:25.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:25.271+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:25.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:25.271+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:25.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:25.272+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:25.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:25.467+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:25.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:25.467+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:25.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:25.467+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:25.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:25.468+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:26.022 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:25.534+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:26.022 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:25.534+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:26.022 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:25.534+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:26.022 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:25.535+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:26.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:26.496+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:26.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:26.496+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:26.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:26.496+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:26.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:26.497+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:26.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:26.305+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:26.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:26.305+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:26.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:26.305+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:26.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:26.306+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:26.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:26.438+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:26.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:26.438+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:26.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:26.438+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:26.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:26.439+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:27.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:27.439+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:27.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:27.439+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:27.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:27.439+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:27.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:27.439+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:27.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:27.316+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:27.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:27.316+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:27.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:27.316+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:27.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:27.317+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:27.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:27.514+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:27.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:27.514+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:27.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:27.514+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:27.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:27.514+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:28.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:28.539+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:28.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:28.539+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:28.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:28.539+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:28.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:28.540+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:28.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:28.298+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:28.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:28.298+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:28.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:28.298+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:28.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:28.299+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:28.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:28.463+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:28.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:28.463+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:28.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:28.463+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:28.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:28.464+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:29.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:12:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:12:29.405+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 6590 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:12:29.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:29.258+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:29.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:29.258+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:29.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:29.258+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:29.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:29.258+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:29.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:29.494+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:29.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:29.494+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:29.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:29.494+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:29.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:29.495+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:29.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:29.475+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:29.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:29.475+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:29.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:29.475+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:29.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:29.475+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:30.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:30.306+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:30.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:30.306+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:30.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:30.306+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:30.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:30.307+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:30.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:30.452+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:30.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:30.452+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:30.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:30.452+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:30.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:30.453+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:30.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:30.450+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:30.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:30.450+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:30.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:30.450+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:30.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:30.451+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:31.663 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:31.295+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:31.663 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:31.295+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:31.663 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:31.295+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:31.664 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:31.295+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:31.664 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:31.412+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:31.664 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:31.412+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:31.664 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:31.412+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:31.664 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:31.413+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:31.664 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:31.495+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:31.664 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:31.495+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:31.665 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:31.495+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:31.665 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:31.496+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:32.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:32.313+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:32.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:32.313+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:32.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:32.313+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:32.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:32.314+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:32.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:32.396+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:32.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:32.396+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:32.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:32.396+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:32.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:32.397+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:33.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:32.537+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:33.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:32.537+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:33.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:32.537+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:33.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:32.538+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:33.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:33.337+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:33.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:33.337+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:33.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:33.337+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:33.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:33.337+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:33.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:33.432+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:33.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:33.432+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:33.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:33.432+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:33.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:33.433+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:33.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:33.580+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:33.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:33.580+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:33.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:33.580+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:33.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:33.581+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:34.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:12:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:12:34.406+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 6684 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:12:34.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:34.388+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:34.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:34.388+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:34.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:34.388+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:34.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:34.388+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:34.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:34.348+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:34.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:34.348+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:34.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:34.348+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:34.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:34.348+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:34.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:34.575+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:34.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:34.575+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:34.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:34.575+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:34.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:34.576+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:35.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:12:35.076+0000 7f54d75ae700 0 monclient(hunting): authenticate timed out after 300 2024-03-11T23:12:35.082 INFO:teuthology.orchestra.run.smithi153.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-03-11T23:12:35.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:35.339+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:35.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:35.339+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:35.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:35.339+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:35.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:35.340+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:35.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:35.585+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:35.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:35.585+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:35.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:35.585+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:35.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:35.586+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:35.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:35.424+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:35.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:35.424+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:35.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:35.424+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:35.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:35.425+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:35.823 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-03-11T23:12:36.091 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/mon.smithi153/config 2024-03-11T23:12:36.520 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:36.390+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:36.520 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:36.390+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:36.520 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:36.390+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:36.520 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:36.391+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:36.521 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:36.321+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:36.521 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:36.321+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:36.521 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:36.321+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:36.521 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:36.322+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:36.782 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:36.587+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:36.782 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:36.587+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:36.782 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:36.587+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:36.782 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:36.588+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:37.618 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:37.349+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:37.618 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:37.349+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:37.618 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:37.349+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:37.618 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:37.350+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:37.619 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:37.403+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:37.619 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:37.403+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:37.619 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:37.403+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:37.619 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:37.404+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:37.914 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:37.614+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:37.914 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:37.614+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:37.914 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:37.614+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:37.914 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:37.615+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:38.620 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:38.332+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:38.620 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:38.332+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:38.620 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:38.332+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:38.620 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:38.332+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:38.621 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:38.396+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:38.621 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:38.396+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:38.621 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:38.396+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:38.621 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:38.397+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:39.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:38.616+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:39.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:38.616+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:39.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:38.616+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:39.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:38.617+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:39.779 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:12:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:12:39.408+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 6776 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:12:39.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:39.574+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:39.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:39.574+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:39.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:39.574+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:39.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:39.575+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:39.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:39.407+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:39.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:39.407+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:39.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:39.407+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:39.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:39.408+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:39.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:39.371+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:39.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:39.371+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:39.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:39.371+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:39.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:39.372+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:40.603 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:40.384+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:40.603 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:40.384+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:40.603 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:40.384+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:40.603 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:40.385+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:40.604 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:40.394+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:40.604 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:40.394+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:40.604 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:40.394+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:40.604 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:40.395+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:41.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:40.599+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:41.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:40.599+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:41.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:40.599+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:41.031 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:40.600+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:41.664 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:41.578+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:41.664 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:41.578+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:41.664 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:41.578+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:41.664 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:41.578+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:41.665 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:41.365+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:41.665 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:41.365+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:41.665 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:41.365+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:41.666 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:41.366+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:41.666 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:41.395+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:41.666 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:41.395+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:41.666 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:41.395+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:41.667 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:41.395+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:42.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:42.571+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:42.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:42.571+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:42.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:42.571+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:42.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:42.572+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:42.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:42.376+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:42.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:42.376+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:42.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:42.376+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:42.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:42.377+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:42.782 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:42.436+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:42.782 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:42.436+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:42.782 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:42.436+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:42.782 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:42.437+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:43.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:43.541+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:43.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:43.541+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:43.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:43.541+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:43.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:43.542+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:43.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:43.330+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:43.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:43.330+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:43.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:43.330+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:43.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:43.331+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:43.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:43.407+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:43.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:43.407+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:43.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:43.407+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:43.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:43.407+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:44.558 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:12:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:12:44.409+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 6871 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:12:44.558 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:44.431+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:44.558 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:44.431+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:44.558 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:44.431+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:44.559 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:44.432+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:44.559 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:44.293+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:44.559 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:44.293+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:44.559 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:44.293+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:44.559 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:44.294+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:45.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:44.554+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:45.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:44.554+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:45.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:44.554+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:45.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:44.555+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:45.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:45.341+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:45.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:45.341+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:45.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:45.341+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:45.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:45.342+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:45.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:45.422+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:45.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:45.422+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:45.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:45.422+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:45.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:45.423+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:45.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:45.575+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:45.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:45.575+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:45.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:45.575+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:45.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:45.576+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:46.623 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:46.363+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:46.624 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:46.363+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:46.624 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:46.363+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:46.624 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:46.364+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:46.624 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:46.419+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:14.678584+0000 front 2024-03-11T23:02:14.678594+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:46.624 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:46.419+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:14.678662+0000 front 2024-03-11T23:02:14.678653+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:46.624 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:46.419+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:14.678619+0000 front 2024-03-11T23:02:14.678677+0000 (oldest deadline 2024-03-11T23:02:40.578505+0000) 2024-03-11T23:12:46.624 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:46.419+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:47.029 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:46.619+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:47.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:46.619+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:47.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:46.619+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:47.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:46.620+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:47.646 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:47.436+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:47.647 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:47.347+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:47.647 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:47.347+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:47.647 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:47.347+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:47.647 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:47.348+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:48.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:47.642+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:48.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:47.642+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:48.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:47.642+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:48.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:47.643+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:48.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:48.477+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:48.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:48.391+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:48.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:48.391+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:48.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:48.391+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:48.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:48.391+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:48.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:48.620+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:48.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:48.620+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:48.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:48.620+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:48.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:48.621+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:49.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:12:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:12:49.410+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 6964 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:12:49.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:49.476+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:49.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:49.384+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:49.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:49.384+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:49.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:49.384+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:49.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:49.385+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:49.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:49.629+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:49.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:49.629+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:49.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:49.629+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:49.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:49.630+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:50.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:50.443+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:50.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:50.616+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:50.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:50.616+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:50.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:50.616+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:50.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:50.617+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:50.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:50.382+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:50.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:50.382+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:50.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:50.382+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:50.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:50.383+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:51.645 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:51.444+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:51.645 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:51.363+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:51.645 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:51.363+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:51.645 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:51.363+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:51.646 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:51.363+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:52.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:51.641+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:52.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:51.641+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:52.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:51.641+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:52.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:51.642+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:52.692 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:52.445+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:52.693 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:52.379+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:52.693 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:52.379+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:52.693 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:52.379+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:52.693 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:52.380+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:53.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:52.688+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:53.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:52.688+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:53.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:52.688+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:53.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:52.689+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:53.654 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:53.419+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:53.654 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:53.347+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:53.654 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:53.347+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:53.654 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:53.347+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:53.654 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:53.348+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:54.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:53.650+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:54.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:53.650+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:54.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:53.650+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:54.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:53.650+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:54.675 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:12:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:12:54.411+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 7059 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:12:54.675 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:54.405+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:54.676 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:54.334+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:54.676 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:54.334+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:54.676 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:54.334+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:54.676 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:54.334+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:55.029 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:54.671+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.593118+0000 front 2024-03-11T23:02:19.593080+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:55.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:54.671+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.593057+0000 front 2024-03-11T23:02:19.593098+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:55.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:54.671+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.593144+0000 front 2024-03-11T23:02:19.593066+0000 (oldest deadline 2024-03-11T23:02:44.893024+0000) 2024-03-11T23:12:55.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:54.671+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:55.699 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:55.381+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:55.700 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:55.327+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:55.700 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:55.327+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:55.700 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:55.327+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:55.700 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:55.327+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:56.029 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:55.695+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:56.693 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:56.339+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:56.693 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:56.345+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:56.693 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:56.345+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:56.694 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:56.345+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:56.694 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:56.346+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:57.029 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:56.689+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:57.734 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:57.363+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:57.735 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:57.348+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:57.735 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:57.348+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:57.735 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:57.348+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:57.735 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:57.350+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:58.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:57.730+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:58.718 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:58.352+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:58.718 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:58.354+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:58.719 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:58.354+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:58.719 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:58.354+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:58.719 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:58.355+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:59.029 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:58.714+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:59.748 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:12:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:12:59.358+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:12:59.748 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:12:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:12:59.412+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 7152 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:12:59.749 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:59.360+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 since back 2024-03-11T23:02:19.685498+0000 front 2024-03-11T23:02:19.685459+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:59.749 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:59.360+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 since back 2024-03-11T23:02:19.685432+0000 front 2024-03-11T23:02:19.685470+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:59.749 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:59.361+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 since back 2024-03-11T23:02:19.685530+0000 front 2024-03-11T23:02:19.685489+0000 (oldest deadline 2024-03-11T23:02:42.585377+0000) 2024-03-11T23:12:59.749 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:12:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:12:59.361+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:00.029 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:12:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:12:59.744+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:00.724 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:00.412+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:00.725 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:00.402+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:01.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:00.720+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:01.666 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:01.389+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:01.666 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:01.416+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:02.029 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:01.762+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:02.738 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:02.359+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:02.739 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:02.461+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:03.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:02.734+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:03.743 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:03.332+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:03.744 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:03.436+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:04.029 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:03.739+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:04.695 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:04.311+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:04.695 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:04.397+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:04.696 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:13:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:13:04.413+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 7247 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:13:05.029 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:04.691+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:05.649 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:05.326+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:05.649 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:05.407+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:05.958 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:05.645+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:06.660 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:06.356+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:06.660 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:06.390+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:07.029 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:06.656+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:07.635 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:07.323+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:07.635 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:07.387+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:07.635 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:07.388+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:07.635 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:07.388+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:07.635 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:07.388+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:08.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:07.631+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:08.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:08.614+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:08.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:08.367+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:08.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:08.375+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:08.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:08.375+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:08.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:08.375+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:08.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:08.375+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:09.651 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:13:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:13:09.414+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 7340 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:13:09.652 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:09.328+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:09.652 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:09.380+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:09.652 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:09.380+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:09.652 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:09.380+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:09.652 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:09.380+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:10.029 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:09.648+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:10.631 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:10.300+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:10.632 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:10.390+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:10.632 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:10.390+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:10.632 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:10.390+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:10.632 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:10.390+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:11.029 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:10.627+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:11.666 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:11.308+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:11.666 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:11.401+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:11.666 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:11.401+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:11.666 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:11.401+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:11.666 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:11.402+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:12.029 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:11.662+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:12.381 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:12.338+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:12.693 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:12.377+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:12.694 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:12.377+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:12.694 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:12.377+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:12.694 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:12.378+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:13.029 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:12.690+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:13.653 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:13.318+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:13.653 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:13.402+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:13.653 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:13.402+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:13.653 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:13.402+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:13.654 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:13.403+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:14.029 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:13.649+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:14.666 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:13:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:13:14.415+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 7434 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:13:14.666 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:14.352+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:14.666 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:14.364+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:14.666 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:14.364+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:14.666 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:14.364+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:14.667 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:14.365+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:15.029 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:14.661+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:15.529 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:15.394+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:15.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:15.339+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:15.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:15.339+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:15.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:15.339+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:15.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:15.340+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:16.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:15.618+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:16.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:15.618+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:16.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:15.618+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:16.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:15.619+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:16.673 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:16.404+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:16.673 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:16.302+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:16.673 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:16.302+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:16.673 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:16.302+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:16.673 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:16.303+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:17.029 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:16.669+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:17.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:16.669+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:17.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:16.669+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:17.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:16.669+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:17.689 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:17.451+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:17.689 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:17.302+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:17.689 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:17.302+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:17.689 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:17.302+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:17.690 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:17.303+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:18.029 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:17.685+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:18.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:17.685+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:18.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:17.685+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:18.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:17.686+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:18.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:18.415+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:18.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:18.299+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:18.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:18.299+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:18.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:18.299+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:18.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:18.300+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:19.029 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:18.686+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:19.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:18.686+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:19.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:18.686+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:19.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:18.686+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:19.671 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:19.399+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:19.672 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:13:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:13:19.416+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 7526 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:13:19.672 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:19.318+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:19.672 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:19.318+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:19.672 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:19.318+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:19.672 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:19.319+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:20.029 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:19.667+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:20.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:19.667+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:20.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:19.667+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:20.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:19.668+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:20.646 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:20.357+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:20.647 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:20.354+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:20.647 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:20.354+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:20.647 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:20.354+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:20.647 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:20.356+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:21.029 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:20.643+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:21.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:20.643+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:21.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:20.643+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:21.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:20.644+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:21.667 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:21.363+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:21.667 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:21.363+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:21.667 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:21.363+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:21.668 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:21.364+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:21.668 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:21.320+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:21.668 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:21.320+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:21.668 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:21.320+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:21.668 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:21.321+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:22.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:21.678+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:22.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:21.678+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:22.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:21.678+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:22.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:21.680+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:22.686 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:22.328+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:22.686 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:22.328+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:22.687 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:22.328+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:22.687 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:22.329+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:22.687 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:22.381+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:22.687 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:22.381+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:22.687 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:22.381+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:22.687 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:22.382+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:23.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:22.682+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:23.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:22.682+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:23.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:22.682+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:23.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:22.683+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:23.728 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:23.364+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:23.729 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:23.364+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:23.729 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:23.364+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:23.729 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:23.365+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:23.729 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:23.332+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:23.729 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:23.332+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:23.729 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:23.332+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:23.729 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:23.333+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:24.029 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:23.724+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:24.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:23.724+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:24.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:23.724+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:24.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:23.725+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:24.764 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:13:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:13:24.417+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 7621 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:13:24.765 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:24.346+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:24.765 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:24.346+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:24.765 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:24.346+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:24.765 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:24.346+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:24.765 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:24.389+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:24.765 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:24.389+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:24.765 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:24.389+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:24.765 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:24.390+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:25.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:24.760+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:25.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:24.760+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:25.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:24.760+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:25.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:24.762+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:25.750 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:25.375+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:25.750 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:25.375+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:25.750 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:25.375+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:25.750 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:25.376+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:25.750 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:25.343+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:25.750 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:25.343+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:25.751 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:25.343+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:25.751 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:25.344+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:26.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:25.746+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:26.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:25.746+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:26.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:25.746+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:26.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:25.746+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:26.740 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:26.381+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:26.740 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:26.381+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:26.740 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:26.381+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:26.740 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:26.382+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:26.740 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:26.341+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:26.740 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:26.341+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:26.741 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:26.341+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:26.741 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:26.342+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:27.029 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:26.736+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:27.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:26.736+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:27.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:26.736+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:27.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:26.736+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:27.729 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:27.293+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:27.729 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:27.293+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:27.729 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:27.293+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:27.729 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:27.294+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:27.730 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:27.344+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:27.730 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:27.344+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:27.730 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:27.344+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:27.730 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:27.345+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:28.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:27.725+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:28.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:27.725+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:28.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:27.725+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:28.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:27.726+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:28.748 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:28.309+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:28.748 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:28.309+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:28.748 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:28.309+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:28.748 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:28.310+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:28.749 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:28.347+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:28.749 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:28.347+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:28.749 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:28.347+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:28.749 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:28.348+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:29.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:28.744+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:29.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:28.744+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:29.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:28.744+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:29.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:28.745+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:29.762 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:13:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:13:29.418+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 7714 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:13:29.763 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:29.325+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:29.763 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:29.325+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:29.763 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:29.325+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:29.763 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:29.326+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:29.763 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:29.301+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:29.763 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:29.301+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:29.764 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:29.301+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:29.764 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:29.302+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:30.029 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:29.758+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:30.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:29.758+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:30.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:29.758+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:30.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:29.759+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:30.757 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:30.330+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:30.757 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:30.330+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:30.758 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:30.330+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:30.758 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:30.332+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:30.758 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:30.329+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:30.758 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:30.329+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:30.758 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:30.329+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:30.758 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:30.330+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:31.029 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:30.753+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:31.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:30.753+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:31.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:30.753+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:31.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:30.754+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:31.668 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:31.347+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:31.668 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:31.347+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:31.669 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:31.347+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:31.669 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:31.348+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:31.669 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:31.326+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:31.669 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:31.326+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:31.669 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:31.326+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:31.669 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:31.326+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:32.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:31.751+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:32.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:31.751+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:32.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:31.751+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:32.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:31.751+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:32.711 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:32.329+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:32.711 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:32.329+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:32.711 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:32.329+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:32.711 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:32.330+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:32.711 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:32.280+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:32.711 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:32.280+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:32.711 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:32.280+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:32.711 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:32.280+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:33.029 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:32.707+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:33.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:32.707+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:33.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:32.707+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:33.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:32.708+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:33.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:33.353+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:33.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:33.353+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:33.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:33.353+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:33.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:33.353+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:33.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:33.232+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:33.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:33.232+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:33.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:33.232+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:33.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:33.232+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:34.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:33.752+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:34.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:33.752+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:34.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:33.752+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:34.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:33.753+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:34.529 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:13:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:13:34.419+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 7808 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:13:34.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:34.363+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:34.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:34.363+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:34.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:34.363+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:34.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:34.364+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:34.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:34.256+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:34.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:34.256+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:34.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:34.256+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:34.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:34.256+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:35.029 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:34.735+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:35.065 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:34.735+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:35.065 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:34.735+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:35.066 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:34.735+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:35.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:35.410+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:35.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:35.410+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:35.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:35.410+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:35.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:35.410+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:35.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:35.270+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:35.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:35.270+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:35.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:35.270+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:35.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:35.271+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:36.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:35.690+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:36.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:35.690+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:36.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:35.690+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:36.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:35.691+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:36.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:36.267+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:36.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:36.267+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:36.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:36.267+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:36.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:36.268+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:36.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:36.362+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:36.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:36.362+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:36.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:36.362+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:36.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:36.363+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:37.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:36.724+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:37.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:36.724+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:37.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:36.724+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:37.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:36.725+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:37.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:37.396+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:37.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:37.396+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:37.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:37.396+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:37.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:37.397+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:37.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:37.269+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:37.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:37.269+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:37.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:37.269+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:37.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:37.270+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:38.029 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:37.697+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:38.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:37.697+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:38.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:37.697+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:38.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:37.698+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:38.743 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:38.277+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:38.743 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:38.277+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:38.743 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:38.277+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:38.743 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:38.278+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:38.744 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:38.350+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:38.744 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:38.350+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:38.744 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:38.350+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:38.744 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:38.350+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:39.029 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:38.739+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:39.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:38.739+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:39.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:38.739+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:39.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:38.740+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:39.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:13:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:13:39.420+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 7900 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:13:39.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:39.379+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:39.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:39.379+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:39.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:39.379+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:39.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:39.379+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:39.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:39.258+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:39.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:39.258+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:39.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:39.258+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:39.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:39.259+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:40.225 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:39.784+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:40.226 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:39.784+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:40.226 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:39.784+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:40.226 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:39.785+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:40.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:40.375+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:40.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:40.375+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:40.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:40.375+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:40.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:40.376+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:40.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:40.222+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:40.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:40.222+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:40.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:40.222+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:40.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:40.222+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:41.029 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:40.774+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:41.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:40.774+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:41.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:40.774+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:41.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:40.774+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:41.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:41.378+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:41.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:41.378+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:41.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:41.378+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:41.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:41.379+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:41.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:41.256+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:41.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:41.256+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:41.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:41.256+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:41.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:41.257+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:42.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:41.771+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:42.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:41.771+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:42.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:41.771+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:42.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:41.772+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:42.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:42.388+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:42.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:42.388+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:42.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:42.388+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:42.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:42.389+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:42.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:42.210+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:42.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:42.210+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:42.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:42.210+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:42.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:42.211+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:43.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:42.722+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:43.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:42.722+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:43.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:42.722+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:43.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:42.723+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:43.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:43.406+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:43.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:43.406+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:43.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:43.406+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:43.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:43.406+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:43.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:43.255+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:43.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:43.255+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:43.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:43.255+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:43.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:43.256+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:44.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:43.772+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:44.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:43.772+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:44.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:43.772+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:44.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:43.773+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:44.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:13:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:13:44.421+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 7995 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:13:44.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:44.411+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:44.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:44.411+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:44.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:44.411+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:44.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:44.412+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:44.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:44.268+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:44.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:44.268+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:44.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:44.268+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:44.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:44.269+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:45.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:44.810+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:45.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:44.810+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:45.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:44.810+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:45.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:44.811+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:45.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:45.455+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:45.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:45.455+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:45.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:45.455+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:45.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:45.456+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:45.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:45.271+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:45.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:45.271+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:45.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:45.271+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:45.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:45.272+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:46.240 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:45.781+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:46.240 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:45.781+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:46.240 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:45.781+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:46.240 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:45.782+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:46.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:46.450+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:46.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:46.450+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:46.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:46.450+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:46.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:46.450+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:46.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:46.236+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:46.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:46.236+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:46.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:46.236+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:46.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:46.237+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:47.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:46.772+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:47.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:46.772+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:47.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:46.772+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:47.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:46.773+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:47.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:47.422+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:47.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:47.422+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:47.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:47.422+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:47.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:47.423+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:47.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:47.255+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:47.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:47.255+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:47.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:47.255+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:47.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:47.256+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:48.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:47.787+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:48.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:47.787+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:48.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:47.787+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:48.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:47.788+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:48.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:48.298+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:48.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:48.298+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:48.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:48.298+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:48.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:48.299+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:48.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:48.400+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:48.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:48.400+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:48.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:48.400+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:48.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:48.401+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:49.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:48.824+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:49.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:48.824+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:49.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:48.824+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:49.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:48.825+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:49.554 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:13:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:13:49.423+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 8089 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:13:49.554 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:49.299+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:49.554 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:49.299+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:49.554 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:49.299+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:49.554 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:49.300+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:49.554 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:49.372+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:49.555 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:49.372+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:49.555 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:49.372+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:49.555 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:49.373+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:50.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:49.779+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:50.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:49.779+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:50.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:49.779+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:50.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:49.779+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:50.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:50.317+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:50.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:50.317+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:50.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:50.317+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:50.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:50.318+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:50.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:50.327+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:50.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:50.327+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:50.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:50.327+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:50.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:50.328+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:51.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:50.783+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:51.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:50.783+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:51.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:50.783+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:51.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:50.784+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:51.670 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:51.311+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:51.670 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:51.311+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:51.670 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:51.311+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:51.670 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:51.312+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:51.670 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:51.319+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:51.670 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:51.319+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:51.670 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:51.319+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:51.670 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:51.319+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:52.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:51.798+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:52.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:51.798+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:52.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:51.798+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:52.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:51.798+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:52.288 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:52.263+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:52.288 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:52.263+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:52.289 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:52.263+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:52.289 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:52.264+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:52.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:52.284+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:52.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:52.284+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:52.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:52.284+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:52.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:52.285+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:53.224 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:52.807+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:53.224 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:52.807+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:53.225 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:52.807+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:53.225 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:52.807+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:53.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:53.220+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:53.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:53.220+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:53.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:53.220+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:53.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:53.221+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:53.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:53.321+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:53.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:53.321+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:53.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:53.321+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:53.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:53.322+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:54.249 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:53.838+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:54.249 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:53.838+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:54.249 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:53.838+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:54.249 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:53.838+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:54.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:13:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:13:54.424+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 8184 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:13:54.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:54.245+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:54.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:54.245+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:54.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:54.245+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:54.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:54.245+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:54.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:54.350+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:54.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:54.350+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:54.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:54.350+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:54.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:54.350+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:55.110 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:54.817+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:55.110 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:54.817+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:55.110 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:54.817+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:55.110 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:54.818+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:55.392 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:55.235+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:55.392 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:55.235+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:55.392 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:55.235+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:55.393 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:55.235+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:55.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:55.388+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:55.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:55.388+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:55.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:55.388+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:55.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:55.389+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:56.196 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:55.851+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:56.196 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:55.851+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:56.196 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:55.851+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:56.196 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:55.852+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:56.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:56.192+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:56.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:56.192+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:56.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:56.192+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:56.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:56.193+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:56.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:56.423+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:56.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:56.423+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:56.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:56.423+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:56.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:56.424+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:57.161 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:56.891+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:57.161 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:56.891+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:57.161 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:56.891+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:57.161 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:56.892+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:57.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:57.157+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:57.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:57.157+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:57.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:57.157+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:57.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:57.158+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:57.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:57.381+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:57.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:57.381+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:57.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:57.381+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:57.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:57.381+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:58.177 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:57.912+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:58.178 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:57.912+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:58.178 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:57.912+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:58.178 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:57.912+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:58.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:58.173+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:58.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:58.173+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:58.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:58.173+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:58.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:58.174+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:58.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:58.394+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:58.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:58.394+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:58.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:58.394+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:58.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:58.395+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:59.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:58.906+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:59.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:58.906+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:59.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:58.906+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:13:59.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:58.907+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:59.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:59.152+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:59.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:59.152+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:59.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:59.152+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:13:59.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:13:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:13:59.153+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:13:59.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:13:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:13:59.425+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 8277 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:13:59.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:59.434+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:59.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:59.434+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:59.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:59.435+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:13:59.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:13:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:13:59.436+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:00.182 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:59.882+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:00.182 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:59.882+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:00.183 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:59.882+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:00.183 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:13:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:13:59.883+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:00.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:00.178+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:00.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:00.178+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:00.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:00.178+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:00.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:00.179+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:00.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:00.395+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:00.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:00.395+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:00.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:00.395+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:00.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:00.396+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:01.170 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:00.912+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:01.170 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:00.912+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:01.170 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:00.912+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:01.170 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:00.913+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:01.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:01.358+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:01.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:01.358+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:01.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:01.358+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:01.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:01.359+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:01.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:01.166+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:01.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:01.166+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:01.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:01.166+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:01.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:01.167+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:02.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:01.890+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:02.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:01.890+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:02.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:01.890+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:02.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:01.891+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:02.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:02.394+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:02.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:02.394+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:02.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:02.394+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:02.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:02.395+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:02.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:02.159+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:02.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:02.159+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:02.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:02.159+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:02.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:02.160+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:03.201 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:02.894+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:03.201 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:02.894+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:03.201 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:02.894+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:03.201 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:02.895+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:03.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:03.197+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:03.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:03.197+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:03.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:03.197+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:03.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:03.198+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:03.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:03.431+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:03.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:03.431+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:03.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:03.431+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:03.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:03.431+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:04.206 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:03.915+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:04.206 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:03.915+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:04.206 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:03.915+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:04.206 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:03.916+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:04.529 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:14:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:14:04.426+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 8373 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:14:04.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:04.202+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:04.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:04.202+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:04.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:04.202+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:04.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:04.203+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:04.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:04.387+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:04.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:04.388+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:04.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:04.388+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:04.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:04.389+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:05.219 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:04.951+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:05.219 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:04.951+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:05.219 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:04.951+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:05.219 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:04.951+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:05.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:05.215+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:05.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:05.215+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:05.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:05.215+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:05.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:05.216+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:05.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:05.409+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:05.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:05.409+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:05.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:05.409+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:05.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:05.410+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:06.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:05.964+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:06.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:05.964+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:06.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:05.964+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:06.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:05.965+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:06.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:06.177+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:06.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:06.177+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:06.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:06.177+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:06.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:06.178+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:06.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:06.437+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:06.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:06.437+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:06.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:06.437+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:06.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:06.438+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:07.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:07.180+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:07.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:07.180+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:07.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:07.180+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:07.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:07.181+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:07.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:07.000+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:07.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:07.000+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:07.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:07.000+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:07.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:07.001+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:07.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:07.435+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:07.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:07.435+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:07.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:07.435+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:07.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:07.436+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:08.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:08.002+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:08.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:08.002+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:08.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:08.002+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:08.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:08.003+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:08.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:08.168+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:08.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:08.168+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:08.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:08.168+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:08.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:08.168+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:08.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:08.476+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:08.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:08.476+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:08.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:08.476+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:08.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:08.477+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:09.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:09.132+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:09.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:09.132+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:09.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:09.132+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:09.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:09.133+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:09.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:09.017+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:09.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:09.017+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:09.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:09.017+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:09.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:09.018+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:09.555 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:14:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:14:09.427+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 8467 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:14:09.555 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:09.492+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:09.555 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:09.492+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:09.556 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:09.492+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:09.556 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:09.493+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:10.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:10.006+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:10.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:10.006+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:10.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:10.006+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:10.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:10.007+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:10.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:10.086+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:10.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:10.086+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:10.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:10.086+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:10.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:10.087+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:10.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:10.456+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:10.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:10.456+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:10.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:10.456+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:10.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:10.456+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:11.132 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:10.999+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:11.132 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:10.999+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:11.132 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:10.999+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:11.132 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:10.999+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:11.430 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:11.128+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:11.430 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:11.128+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:11.430 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:11.128+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:11.430 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:11.129+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:11.779 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:11.426+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:11.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:11.426+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:11.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:11.426+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:11.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:11.427+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:12.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:12.016+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:12.282 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:12.016+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:12.282 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:12.016+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:12.282 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:12.017+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:12.282 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:12.122+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:12.282 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:12.122+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:12.282 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:12.122+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:12.282 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:12.123+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:12.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:12.456+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:12.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:12.456+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:12.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:12.456+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:12.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:12.457+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:13.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:12.994+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:13.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:12.994+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:13.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:12.994+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:13.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:12.995+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:13.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:13.112+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:13.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:13.112+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:13.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:13.112+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:13.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:13.113+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:13.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:13.464+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:13.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:13.464+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:13.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:13.464+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:13.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:13.465+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:14.426 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:14.035+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:14.427 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:14.035+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:14.427 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:14.035+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:14.427 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:14.036+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:14.427 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:14.096+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:14.427 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:14.096+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:14.427 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:14.096+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:14.427 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:14.097+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:14.779 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:14:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:14:14.428+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 8562 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:14:14.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:14.422+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:14.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:14.422+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:14.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:14.422+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:14.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:14.424+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:15.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:15.075+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:15.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:15.075+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:15.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:15.075+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:15.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:15.076+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:15.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:15.023+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:15.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:15.023+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:15.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:15.023+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:15.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:15.023+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:15.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:15.458+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:15.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:15.458+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:15.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:15.458+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:15.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:15.459+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:16.443 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:16.027+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:16.444 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:16.027+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:16.444 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:16.027+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:16.444 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:16.027+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:16.444 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:16.112+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:16.444 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:16.112+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:16.444 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:16.112+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:16.444 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:16.113+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:16.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:16.439+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:16.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:16.439+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:16.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:16.439+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:16.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:16.440+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:17.488 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:17.079+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:17.488 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:17.079+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:17.489 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:17.079+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:17.489 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:17.079+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:17.489 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:17.048+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:17.489 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:17.048+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:17.489 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:17.048+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:17.489 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:17.048+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:17.779 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:17.485+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:17.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:17.485+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:17.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:17.485+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:17.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:17.486+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:18.502 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:18.037+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:18.502 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:18.037+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:18.502 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:18.037+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:18.502 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:18.038+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:18.502 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:18.032+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:18.503 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:18.032+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:18.503 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:18.032+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:18.503 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:18.033+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:18.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:18.498+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:18.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:18.498+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:18.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:18.498+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:18.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:18.499+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:19.388 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:19.068+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:19.388 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:19.068+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:19.388 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:19.068+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:19.388 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:19.069+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:19.388 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:19.052+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:19.388 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:19.052+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:19.388 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:19.052+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:19.389 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:19.053+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:19.686 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:14:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:14:19.429+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 8655 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:14:19.687 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:19.507+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:19.687 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:19.507+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:19.687 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:19.507+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:19.687 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:19.508+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:20.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:20.028+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:20.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:20.028+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:20.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:20.028+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:20.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:20.029+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:20.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:20.009+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:20.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:20.009+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:20.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:20.009+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:20.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:20.010+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:20.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:20.500+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:20.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:20.500+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:20.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:20.500+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:20.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:20.501+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:21.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:21.049+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:21.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:21.049+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:21.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:21.050+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:21.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:21.051+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:21.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:21.072+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:21.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:21.072+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:21.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:21.072+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:21.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:21.073+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:22.019 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:21.529+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:22.019 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:21.529+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:22.019 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:21.529+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:22.020 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:21.530+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:22.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:22.015+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:22.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:22.015+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:22.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:22.015+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:22.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:22.016+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:22.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:22.094+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:22.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:22.094+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:22.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:22.094+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:22.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:22.095+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:23.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:22.540+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:23.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:22.540+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:23.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:22.540+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:23.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:22.540+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:23.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:23.065+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:23.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:23.065+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:23.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:23.065+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:23.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:23.066+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:23.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:23.048+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:23.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:23.048+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:23.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:23.048+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:23.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:23.049+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:24.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:23.551+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:24.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:23.551+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:24.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:23.551+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:24.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:23.552+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:24.435 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:24.070+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:24.435 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:24.070+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:24.435 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:24.070+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:24.435 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:24.071+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:24.435 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:24.064+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:24.435 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:24.064+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:24.435 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:24.064+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:24.435 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:24.065+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:24.779 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:14:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:14:24.430+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 8751 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:14:24.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:24.550+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:24.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:24.550+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:24.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:24.550+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:24.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:24.551+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:25.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:25.024+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:25.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:25.024+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:25.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:25.024+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:25.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:25.025+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:25.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:25.108+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:25.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:25.108+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:25.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:25.108+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:25.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:25.109+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:25.982 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:25.582+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:25.983 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:25.582+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:25.983 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:25.582+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:25.983 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:25.583+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:26.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:25.978+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:26.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:25.978+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:26.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:25.978+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:26.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:25.979+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:26.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:26.098+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:26.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:26.098+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:26.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:26.098+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:26.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:26.098+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:26.993 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:26.564+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:26.993 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:26.564+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:26.993 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:26.564+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:26.993 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:26.565+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:27.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:27.085+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:27.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:27.085+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:27.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:27.085+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:27.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:27.086+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:27.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:26.989+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:27.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:26.989+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:27.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:26.989+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:27.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:26.990+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:27.986 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:27.550+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:27.986 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:27.550+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:27.986 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:27.550+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:27.986 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:27.551+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:28.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:28.095+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:28.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:28.095+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:28.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:28.095+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:28.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:28.096+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:28.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:27.982+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:28.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:27.982+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:28.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:27.982+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:28.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:27.983+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:28.903 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:28.536+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:28.903 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:28.536+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:28.903 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:28.536+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:28.903 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:28.537+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:29.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:29.097+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:29.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:29.097+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:29.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:29.097+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:29.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:29.098+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:29.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:29.013+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:29.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:29.013+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:29.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:29.013+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:29.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:29.014+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:29.779 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:14:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:14:29.431+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 8845 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:14:29.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:29.583+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:29.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:29.583+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:29.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:29.583+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:29.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:29.584+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:30.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:30.054+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:30.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:30.054+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:30.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:30.054+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:30.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:30.055+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:30.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:29.998+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:30.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:29.998+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:30.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:29.998+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:30.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:29.999+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:31.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:30.581+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:31.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:30.581+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:31.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:30.581+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:31.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:30.582+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:31.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:31.054+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:31.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:31.054+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:31.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:31.054+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:31.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:31.054+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:31.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:31.022+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:31.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:31.022+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:31.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:31.022+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:31.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:31.023+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:31.974 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:31.566+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:31.974 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:31.566+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:31.974 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:31.566+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:31.974 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:31.567+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:32.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:32.050+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:32.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:32.050+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:32.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:32.050+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:32.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:32.051+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:32.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:32.027+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:32.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:32.027+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:32.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:32.027+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:32.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:32.028+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:33.021 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:32.578+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:33.021 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:32.578+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:33.021 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:32.578+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:33.022 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:32.579+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:33.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:33.040+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:33.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:33.040+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:33.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:33.040+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:33.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:33.041+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:33.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:33.017+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:33.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:33.017+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:33.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:33.017+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:33.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:33.018+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:34.006 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:33.584+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:34.006 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:33.584+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:34.006 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:33.584+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:34.006 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:33.585+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:34.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:34.002+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:34.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:34.002+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:34.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:34.002+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:34.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:34.003+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:34.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:34.030+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:34.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:34.030+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:34.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:34.030+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:34.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:34.031+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:34.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:14:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:14:34.432+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 8940 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:14:34.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:34.620+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:34.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:34.620+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:34.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:34.620+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:34.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:34.620+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:35.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:35.007+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:35.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:35.007+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:35.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:35.007+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:35.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:35.008+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:35.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:34.990+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:35.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:34.990+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:35.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:34.990+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:35.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:34.991+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:35.962 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:35.618+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:35.962 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:35.618+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:35.963 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:35.618+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:35.963 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:35.618+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:36.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:35.971+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:36.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:35.971+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:36.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:35.971+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:36.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:35.972+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:36.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:35.958+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:36.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:35.958+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:36.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:35.958+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:36.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:35.959+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:36.931 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:36.604+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:36.931 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:36.604+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:36.931 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:36.604+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:36.931 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:36.605+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:37.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:36.960+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:37.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:36.960+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:37.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:36.960+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:37.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:36.961+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:37.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:36.926+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:37.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:36.926+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:37.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:36.927+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:37.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:36.927+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:37.973 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:37.651+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:37.973 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:37.651+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:37.973 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:37.651+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:37.973 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:37.651+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:38.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:37.969+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:38.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:37.969+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:38.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:37.969+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:38.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:37.970+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:38.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:37.975+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:38.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:37.975+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:38.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:37.975+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:38.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:37.976+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:38.997 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:38.943+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:38.998 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:38.943+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:38.998 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:38.943+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:38.998 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:38.943+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:38.998 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:38.701+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:38.998 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:38.701+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:38.998 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:38.701+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:38.998 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:38.701+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:39.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:38.993+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:39.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:38.993+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:39.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:38.993+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:39.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:38.995+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:39.754 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:14:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:14:39.434+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 9032 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:14:40.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:39.747+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:40.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:39.747+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:40.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:39.747+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:40.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:39.748+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:40.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:39.981+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:40.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:39.981+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:40.030 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:39.981+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:40.031 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:39.982+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:40.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:39.907+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:40.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:39.907+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:40.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:39.907+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:40.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:39.908+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:41.012 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:40.703+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:41.012 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:40.703+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:41.012 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:40.703+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:41.012 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:40.704+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:41.013 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:40.943+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:41.013 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:40.943+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:41.013 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:40.943+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:41.013 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:40.944+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:41.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:41.008+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:41.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:41.008+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:41.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:41.008+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:41.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:41.009+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:41.963 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:41.698+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:41.963 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:41.698+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:41.963 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:41.698+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:41.964 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:41.698+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:42.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:42.025+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:42.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:42.025+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:42.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:42.025+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:42.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:42.026+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:42.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:41.959+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:42.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:41.959+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:42.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:41.959+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:42.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:41.960+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:42.986 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:42.720+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:42.986 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:42.720+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:42.986 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:42.720+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:42.987 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:42.721+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:43.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:43.022+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:43.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:43.022+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:43.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:43.022+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:43.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:43.023+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:43.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:42.982+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:43.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:42.982+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:43.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:42.982+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:43.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:42.983+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:43.960 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:43.699+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:43.960 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:43.699+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:43.960 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:43.699+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:43.960 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:43.699+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:44.262 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:44.007+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:44.266 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:44.007+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:44.266 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:44.007+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:44.266 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:44.008+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:44.266 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:43.956+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:44.267 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:43.956+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:44.267 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:43.956+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:44.267 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:43.956+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:44.529 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:14:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:14:44.435+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 9128 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:14:44.948 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:44.709+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:44.948 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:44.709+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:44.948 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:44.709+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:44.948 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:44.709+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:45.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:44.944+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:45.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:44.944+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:45.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:44.944+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:45.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:44.945+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:45.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:45.038+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:45.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:45.038+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:45.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:45.038+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:45.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:45.039+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:46.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:45.934+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:46.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:45.934+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:46.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:45.934+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:46.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:45.935+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:46.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:45.707+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:46.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:45.707+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:46.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:45.707+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:46.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:45.708+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:46.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:46.080+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:46.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:46.080+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:46.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:46.080+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:46.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:46.081+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:47.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:46.751+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:47.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:46.751+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:47.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:46.751+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:47.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:46.752+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:47.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:46.960+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:47.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:46.960+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:47.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:46.960+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:47.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:46.961+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:47.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:47.091+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:47.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:47.091+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:47.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:47.091+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:47.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:47.092+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:48.002 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:47.745+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:48.002 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:47.745+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:48.002 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:47.745+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:48.002 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:47.745+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:48.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:47.997+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:48.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:47.997+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:48.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:47.997+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:48.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:47.998+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:48.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:48.100+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:48.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:48.100+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:48.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:48.100+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:48.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:48.101+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:49.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:48.759+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:49.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:48.759+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:49.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:48.759+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:49.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:48.759+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:49.387 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:49.095+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:49.388 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:49.095+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:49.388 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:49.095+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:49.388 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:49.095+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:49.388 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:49.045+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:49.388 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:49.045+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:49.388 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:49.045+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:49.388 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:49.046+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:49.779 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:14:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:14:49.436+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 9219 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:14:50.071 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:49.788+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:50.071 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:49.788+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:50.071 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:49.788+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:50.071 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:49.789+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:50.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:50.067+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:50.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:50.067+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:50.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:50.067+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:50.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:50.067+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:50.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:50.094+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:50.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:50.094+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:50.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:50.094+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:50.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:50.095+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:51.090 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:50.828+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:51.090 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:50.828+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:51.090 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:50.828+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:51.090 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:50.829+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:51.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:51.086+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:51.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:51.086+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:51.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:51.086+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:51.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:51.087+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:51.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:51.090+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:51.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:51.090+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:51.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:51.090+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:51.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:51.091+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:52.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:51.802+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:52.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:51.802+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:52.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:51.802+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:52.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:51.802+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:52.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:52.045+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:52.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:52.045+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:52.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:52.045+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:52.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:52.046+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:52.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:52.045+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:52.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:52.045+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:52.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:52.045+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:52.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:52.046+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:53.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:52.775+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:53.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:52.775+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:53.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:52.775+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:53.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:52.775+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:53.478 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:53.088+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:53.478 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:53.088+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:53.478 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:53.088+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:53.478 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:53.088+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:53.478 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:53.082+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:53.478 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:53.082+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:53.479 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:53.082+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:53.479 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:53.082+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:54.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:53.752+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:54.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:53.752+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:54.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:53.752+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:54.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:53.753+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:54.441 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:54.093+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:54.441 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:54.093+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:54.441 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:54.093+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:54.441 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:54.094+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:54.442 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:54.093+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:54.442 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:54.093+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:54.442 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:54.093+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:54.442 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:54.094+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:54.779 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:14:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:14:54.437+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 9311 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:14:55.056 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:54.798+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:55.056 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:54.798+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:55.056 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:54.798+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:55.056 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:54.799+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:55.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:55.109+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:55.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:55.109+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:55.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:55.109+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:55.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:55.110+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:55.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:55.051+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:55.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:55.051+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:55.532 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:55.051+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:55.532 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:55.052+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:56.159 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:55.785+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:56.159 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:55.785+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:56.159 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:55.785+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:56.159 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:55.786+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:56.159 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:56.016+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:56.159 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:56.016+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:56.159 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:56.016+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:56.159 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:56.016+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:56.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:56.155+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:56.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:56.155+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:56.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:56.155+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:56.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:56.155+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:57.141 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:57.026+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:57.142 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:57.026+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:57.142 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:57.026+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:57.142 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:57.027+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:57.142 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:56.828+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:57.142 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:56.828+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:57.142 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:56.828+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:57.142 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:56.829+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:57.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:57.137+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:57.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:57.137+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:57.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:57.137+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:57.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:57.138+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:58.172 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:58.064+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:58.172 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:58.064+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:58.172 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:58.064+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:58.172 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:58.065+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:58.172 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:57.844+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:58.172 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:57.844+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:58.172 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:57.844+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:58.172 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:57.845+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:58.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:58.168+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:58.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:58.168+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:58.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:58.168+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:58.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:58.169+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:59.082 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:58.798+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:59.082 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:58.798+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:59.082 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:58.798+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:14:59.082 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:58.799+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:59.387 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:59.181+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:59.388 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:59.181+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:59.388 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:59.181+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:14:59.388 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:14:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:14:59.181+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:59.388 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:59.078+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:59.388 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:59.078+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:59.388 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:59.078+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:14:59.388 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:14:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:14:59.078+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:14:59.779 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:14:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:14:59.438+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 9397 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:15:00.180 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:59.809+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:00.180 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:59.809+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:00.180 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:59.809+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:00.180 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:14:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:14:59.810+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:00.181 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:00.047+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:00.181 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:00.047+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:00.181 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:00.047+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:00.181 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:00.047+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:00.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:00.176+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:00.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:00.176+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:00.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:00.176+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:00.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:00.177+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:01.160 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:00.786+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:01.160 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:00.786+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:01.160 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:00.786+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:01.160 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:00.787+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:01.161 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:01.031+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:01.161 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:01.031+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:01.161 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:01.031+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:01.161 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:01.032+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:01.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:01.156+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:01.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:01.156+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:01.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:01.156+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:01.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:01.157+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:02.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:01.794+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:02.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:01.794+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:02.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:01.794+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:02.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:01.795+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:02.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:02.118+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:02.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:02.118+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:02.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:02.118+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:02.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:02.119+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:02.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:02.076+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:02.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:02.076+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:02.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:02.076+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:02.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:02.077+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:03.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:02.828+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:03.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:02.828+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:03.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:02.828+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:03.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:02.830+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:03.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:03.110+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:03.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:03.110+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:03.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:03.110+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:03.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:03.111+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:03.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:03.115+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:03.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:03.115+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:03.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:03.115+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:03.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:03.116+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:04.097 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:03.832+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:04.097 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:03.832+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:04.098 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:03.832+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:04.098 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:03.833+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:04.444 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:04.093+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:04.444 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:04.093+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:04.444 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:04.093+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:04.444 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:04.094+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:04.444 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:04.150+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:04.444 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:04.150+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:04.445 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:04.150+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:04.445 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:04.151+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:04.781 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:15:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:15:04.439+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 9483 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:15:05.137 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:05.058+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:05.138 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:05.058+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:05.138 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:05.058+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:05.138 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:05.059+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:05.138 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:04.881+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:05.138 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:04.881+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:05.138 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:04.881+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:05.138 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:04.882+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:05.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:05.133+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:05.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:05.133+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:05.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:05.133+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:05.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:05.134+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:06.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:05.912+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:06.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:05.912+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:06.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:05.912+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:06.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:05.913+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:06.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:06.064+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:06.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:06.064+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:06.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:06.064+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:06.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:06.065+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:06.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:06.161+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:06.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:06.161+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:06.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:06.161+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:06.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:06.162+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:07.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:07.149+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:07.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:07.149+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:07.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:07.149+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:07.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:07.150+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:07.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:07.077+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:07.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:07.077+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:07.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:07.077+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:07.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:07.077+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:07.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:06.955+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:07.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:06.955+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:07.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:06.955+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:07.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:06.956+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:08.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:08.183+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:08.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:08.183+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:08.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:08.183+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:08.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:08.184+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:08.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:08.109+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:08.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:08.109+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:08.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:08.109+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:08.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:08.110+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:08.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:07.974+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:08.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:07.974+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:08.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:07.974+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:08.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:07.974+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:09.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:09.136+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:09.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:09.136+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:09.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:09.136+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:09.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:09.137+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:09.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:09.091+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:09.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:09.091+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:09.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:09.091+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:09.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:09.092+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:09.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:09.008+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:09.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:09.008+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:09.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:09.008+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:09.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:09.009+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:09.779 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:15:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:15:09.441+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 9570 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:15:10.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:10.089+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:10.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:10.089+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:10.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:10.089+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:10.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:10.090+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:10.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:10.076+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:10.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:10.076+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:10.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:10.076+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:10.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:10.077+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:10.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:10.011+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:10.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:10.011+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:10.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:10.011+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:10.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:10.012+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:11.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:11.040+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:11.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:11.040+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:11.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:11.040+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:11.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:11.041+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:11.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:11.111+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:11.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:11.111+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:11.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:11.111+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:11.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:11.112+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:11.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:11.055+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:11.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:11.055+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:11.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:11.055+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:11.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:11.056+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:12.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:12.093+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:12.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:12.093+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:12.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:12.093+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:12.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:12.094+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:12.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:12.032+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:12.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:12.032+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:12.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:12.032+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:12.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:12.033+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:12.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:12.063+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:12.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:12.063+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:12.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:12.063+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:12.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:12.064+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:13.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:13.135+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:13.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:13.135+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:13.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:13.135+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:13.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:13.136+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:13.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:13.050+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:13.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:13.050+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:13.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:13.050+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:13.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:13.051+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:13.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:13.023+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:13.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:13.023+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:13.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:13.023+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:13.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:13.024+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:14.446 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:14.150+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:14.446 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:14.150+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:14.446 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:14.150+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:14.446 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:14.150+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:14.447 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:14.042+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:14.447 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:14.042+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:14.447 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:14.042+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:14.447 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:14.042+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:14.447 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:14.039+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:14.447 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:14.039+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:14.447 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:14.039+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:14.447 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:14.040+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:14.779 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:15:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:15:14.442+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 9658 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:15:15.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:15.054+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:15.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:15.054+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:15.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:15.054+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:15.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:15.054+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:15.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:15.170+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:15.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:15.170+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:15.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:15.170+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:15.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:15.171+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:15.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:15.017+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:15.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:15.017+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:15.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:15.017+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:15.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:15.018+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:16.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:16.025+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:16.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:16.025+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:16.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:16.025+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:16.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:16.026+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:16.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:15.972+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:16.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:15.972+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:16.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:15.972+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:16.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:15.973+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:16.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:16.216+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:16.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:16.216+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:16.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:16.216+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:16.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:16.217+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:17.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:16.988+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:17.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:16.988+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:17.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:16.988+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:17.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:16.988+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:17.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:16.999+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:17.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:16.999+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:17.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:16.999+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:17.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:16.999+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:17.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:17.171+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:17.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:17.171+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:17.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:17.171+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:17.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:17.171+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:18.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:17.999+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:18.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:17.999+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:18.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:17.999+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:18.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:18.000+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:18.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:18.146+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:18.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:18.146+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:18.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:18.146+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:18.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:18.146+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:18.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:17.972+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:18.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:17.972+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:18.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:17.972+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:18.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:17.973+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:19.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:19.128+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:19.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:19.128+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:19.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:19.128+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:19.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:19.129+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:19.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:18.960+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:19.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:18.960+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:19.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:18.960+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:19.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:18.961+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:19.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:18.973+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:19.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:18.973+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:19.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:18.973+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:19.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:18.974+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:19.779 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:15:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:15:19.443+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 9747 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:15:20.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:20.120+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:20.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:20.120+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:20.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:20.120+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:20.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:20.121+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:20.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:19.914+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:20.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:19.914+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:20.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:19.914+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:20.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:19.915+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:20.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:20.008+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:20.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:20.008+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:20.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:20.008+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:20.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:20.009+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:21.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:21.113+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:21.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:21.113+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:21.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:21.113+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:21.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:21.113+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:21.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:20.933+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:21.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:20.933+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:21.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:20.933+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:21.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:20.933+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:21.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:21.056+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:21.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:21.056+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:21.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:21.056+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:21.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:21.057+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:22.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:22.079+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:22.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:22.079+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:22.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:22.079+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:22.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:22.080+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:22.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:21.957+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:22.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:21.957+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:22.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:21.957+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:22.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:21.958+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:22.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:22.065+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:22.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:22.065+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:22.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:22.065+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:22.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:22.066+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:23.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:23.076+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:23.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:23.076+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:23.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:23.076+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:23.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:23.076+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:23.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:22.934+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:23.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:22.934+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:23.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:22.934+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:23.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:22.935+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:23.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:23.021+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:23.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:23.021+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:23.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:23.021+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:23.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:23.022+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:24.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:24.122+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:24.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:24.122+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:24.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:24.122+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:24.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:24.123+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:24.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:23.947+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:24.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:23.947+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:24.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:23.947+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:24.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:23.948+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:24.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:24.022+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:24.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:24.022+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:24.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:24.022+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:24.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:24.023+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:24.779 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:15:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:15:24.444+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 9842 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:15:25.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:25.105+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:25.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:25.105+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:25.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:25.105+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:25.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:25.106+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:25.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:24.981+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:25.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:24.981+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:25.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:24.981+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:25.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:24.982+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:25.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:24.925+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:25.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:24.925+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:25.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:24.925+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:25.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:24.925+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:26.157 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:25.938+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:26.157 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:25.938+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:26.157 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:25.938+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:26.157 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:25.939+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:26.158 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:25.893+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:26.158 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:25.893+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:26.158 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:25.893+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:26.158 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:25.895+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:26.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:26.153+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:26.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:26.153+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:26.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:26.153+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:26.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:26.153+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:27.162 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:26.977+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:27.163 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:26.977+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:27.163 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:26.977+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:27.163 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:26.978+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:27.163 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:26.898+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:27.163 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:26.898+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:27.163 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:26.898+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:27.163 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:26.899+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:27.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:27.158+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:27.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:27.158+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:27.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:27.158+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:27.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:27.159+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:28.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:27.890+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:28.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:27.890+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:28.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:27.890+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:28.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:27.891+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:28.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:28.132+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:28.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:28.132+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:28.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:28.132+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:28.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:28.133+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:28.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:27.976+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:28.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:27.976+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:28.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:27.976+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:28.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:27.977+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:29.138 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:28.872+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:29.139 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:28.872+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:29.139 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:28.872+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:29.139 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:28.873+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:29.139 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:29.012+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:29.139 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:29.012+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:29.139 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:29.012+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:29.139 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:29.012+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:29.450 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:29.134+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:29.450 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:29.134+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:29.450 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:29.134+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:29.451 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:29.135+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:29.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:15:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:15:29.446+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 9935 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:15:30.180 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:29.977+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:30.180 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:29.977+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:30.181 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:29.977+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:30.181 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:29.978+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:30.181 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:29.921+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:30.181 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:29.921+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:30.181 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:29.921+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:30.181 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:29.921+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:30.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:30.176+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:30.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:30.176+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:30.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:30.176+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:30.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:30.176+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:31.158 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:30.883+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:31.159 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:30.883+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:31.159 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:30.883+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:31.159 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:30.884+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:31.159 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:30.989+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:31.159 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:30.989+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:31.159 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:30.989+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:31.159 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:30.990+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:31.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:31.154+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:31.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:31.154+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:31.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:31.154+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:31.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:31.155+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:32.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:31.840+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:32.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:31.840+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:32.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:31.840+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:32.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:31.841+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:32.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:32.162+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:32.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:32.162+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:32.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:32.162+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:32.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:32.164+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:32.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:32.033+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:32.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:32.033+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:32.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:32.033+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:32.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:32.034+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:33.151 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:32.993+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:33.151 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:32.993+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:33.151 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:32.993+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:33.151 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:32.993+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:33.151 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:32.832+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:33.151 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:32.832+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:33.151 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:32.832+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:33.152 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:32.832+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:33.414 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:33.147+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:33.414 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:33.147+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:33.414 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:33.147+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:33.414 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:33.148+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:34.106 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:33.976+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:34.106 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:33.976+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:34.106 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:33.976+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:34.106 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:33.977+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:34.106 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:33.792+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:34.106 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:33.792+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:34.106 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:33.792+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:34.106 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:33.794+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:34.451 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:34.101+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:34.452 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:34.101+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:34.452 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:34.101+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:34.452 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:34.102+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:34.749 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:15:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:15:34.447+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 10028 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:15:35.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:34.746+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:35.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:34.746+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:35.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:34.746+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:35.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:34.746+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:35.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:34.964+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:35.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:34.964+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:35.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:34.964+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:35.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:34.965+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:35.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:35.093+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:35.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:35.093+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:35.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:35.093+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:35.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:35.094+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:36.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:35.921+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:36.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:35.921+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:36.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:35.921+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:36.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:35.921+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:36.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:35.697+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:36.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:35.697+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:36.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:35.697+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:36.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:35.698+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:36.486 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:36.059+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:36.486 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:36.059+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:36.486 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:36.059+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:36.486 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:36.060+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:36.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:36.724+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:36.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:36.724+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:36.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:36.724+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:36.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:36.725+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:37.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:37.044+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:37.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:37.044+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:37.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:37.044+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:37.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:37.045+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:37.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:36.896+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:37.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:36.896+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:37.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:36.896+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:37.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:36.897+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:38.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:37.771+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:38.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:37.771+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:38.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:37.771+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:38.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:37.772+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:38.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:37.859+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:38.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:37.859+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:38.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:37.859+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:38.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:37.860+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:38.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:38.076+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:38.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:38.076+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:38.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:38.076+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:38.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:38.077+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:39.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:38.775+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:39.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:38.775+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:39.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:38.775+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:39.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:38.776+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:39.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:38.873+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:39.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:38.873+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:39.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:38.873+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:39.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:38.874+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:39.388 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:39.048+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:39.388 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:39.048+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:39.388 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:39.048+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:39.388 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:39.049+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:39.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:15:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:15:39.448+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 10121 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:15:40.101 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:39.906+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:40.101 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:39.906+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:40.101 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:39.906+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:40.101 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:39.907+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:40.101 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:39.792+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:40.102 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:39.792+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:40.102 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:39.792+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:40.102 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:39.793+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:40.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:40.097+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:40.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:40.097+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:40.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:40.097+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:40.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:40.097+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:41.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:40.941+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:41.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:40.941+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:41.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:40.941+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:41.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:40.942+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:41.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:40.760+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:41.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:40.760+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:41.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:40.760+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:41.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:40.761+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:41.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:41.092+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:41.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:41.092+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:41.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:41.092+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:41.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:41.093+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:41.978 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:41.808+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:41.978 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:41.808+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:41.978 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:41.808+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:41.978 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:41.808+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:42.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:42.137+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:42.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:42.137+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:42.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:42.137+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:42.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:42.138+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:42.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:41.974+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:42.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:41.974+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:42.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:41.974+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:42.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:41.975+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:43.009 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:42.839+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:43.009 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:42.839+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:43.010 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:42.839+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:43.010 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:42.840+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:43.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:43.185+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:43.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:43.185+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:43.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:43.185+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:43.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:43.186+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:43.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:43.005+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:43.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:43.005+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:43.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:43.005+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:43.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:43.006+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:44.141 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:43.887+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:44.142 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:43.887+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:44.142 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:43.887+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:44.142 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:43.888+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:44.142 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:44.137+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:44.142 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:44.137+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:44.142 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:44.137+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:44.142 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:44.138+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:44.143 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:44.035+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:44.143 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:44.035+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:44.143 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:44.035+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:44.143 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:44.036+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:44.779 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:15:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:15:44.450+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 10214 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:15:45.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:45.018+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:45.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:45.018+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:45.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:45.018+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:45.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:45.018+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:45.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:45.092+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:45.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:45.092+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:45.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:45.092+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:45.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:45.093+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:45.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:44.886+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:45.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:44.886+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:45.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:44.886+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:45.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:44.887+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:45.987 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:45.869+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:45.987 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:45.869+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:45.987 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:45.869+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:45.987 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:45.870+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:46.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:46.103+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:46.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:46.103+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:46.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:46.103+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:46.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:46.104+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:46.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:45.983+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:46.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:45.983+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:46.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:45.983+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:46.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:45.984+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:47.151 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:46.827+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:47.151 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:46.827+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:47.151 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:46.827+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:47.151 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:46.827+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:47.151 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:47.010+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:47.152 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:47.010+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:47.152 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:47.010+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:47.152 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:47.010+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:47.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:47.147+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:47.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:47.147+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:47.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:47.147+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:47.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:47.147+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:48.155 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:48.003+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:48.155 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:48.003+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:48.155 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:48.003+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:48.155 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:48.003+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:48.156 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:47.799+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:48.156 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:47.799+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:48.156 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:47.799+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:48.156 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:47.800+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:48.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:48.145+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:48.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:48.145+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:48.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:48.145+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:48.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:48.146+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:49.026 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:48.768+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:49.026 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:48.768+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:49.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:48.768+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:49.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:48.769+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:49.282 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:49.156+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:49.282 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:49.156+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:49.282 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:49.156+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:49.282 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:49.157+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:49.282 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:49.022+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:49.282 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:49.022+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:49.282 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:49.022+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:49.282 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:49.023+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:49.762 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:15:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:15:49.451+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 10306 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:15:50.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:49.758+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:50.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:49.758+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:50.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:49.758+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:50.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:49.759+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:50.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:49.984+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:50.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:49.984+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:50.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:49.984+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:50.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:49.985+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:50.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:50.197+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:50.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:50.197+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:50.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:50.197+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:50.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:50.198+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:51.012 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:50.736+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:51.012 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:50.736+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:51.012 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:50.736+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:51.012 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:50.737+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:51.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:51.246+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:51.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:51.246+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:51.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:51.246+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:51.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:51.247+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:51.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:51.008+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:51.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:51.008+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:51.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:51.008+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:51.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:51.009+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:52.008 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:51.687+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:52.008 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:51.687+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:52.008 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:51.687+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:52.008 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:51.688+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:52.264 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:52.004+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:52.264 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:52.004+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:52.264 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:52.004+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:52.264 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:52.004+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:52.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:52.260+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:52.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:52.260+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:52.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:52.260+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:52.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:52.261+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:53.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:52.655+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:53.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:52.655+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:53.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:52.655+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:53.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:52.656+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:53.304 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:53.034+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:53.304 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:53.034+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:53.304 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:53.034+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:53.304 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:53.035+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:53.563 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:53.300+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:53.563 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:53.300+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:53.564 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:53.300+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:53.564 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:53.301+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:54.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:53.610+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:54.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:53.610+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:54.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:53.610+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:54.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:53.611+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:54.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:54.258+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:54.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:54.258+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:54.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:54.258+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:54.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:54.259+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:54.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:54.025+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:54.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:54.025+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:54.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:54.025+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:54.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:54.026+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:54.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:15:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:15:54.452+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 10398 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:15:54.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:54.656+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:54.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:54.656+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:54.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:54.656+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:54.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:54.657+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:55.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:55.044+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:55.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:55.044+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:55.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:55.044+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:55.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:55.045+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:55.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:55.272+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:55.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:55.272+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:55.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:55.272+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:55.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:55.273+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:56.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:55.672+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:56.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:55.672+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:56.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:55.672+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:56.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:55.673+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:56.294 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:56.026+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:56.294 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:56.026+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:56.294 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:56.026+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:56.294 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:56.027+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:56.650 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:56.290+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:56.651 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:56.290+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:56.651 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:56.290+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:56.651 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:56.291+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:56.995 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:56.646+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:56.995 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:56.646+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:56.995 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:56.646+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:56.995 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:56.647+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:57.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:56.991+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:57.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:56.991+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:57.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:56.991+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:57.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:56.992+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:57.649 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:57.279+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:57.650 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:57.279+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:57.650 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:57.279+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:57.650 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:57.280+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:57.949 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:57.645+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:57.949 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:57.645+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:57.949 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:57.645+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:57.949 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:57.646+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:58.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:57.945+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:58.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:57.945+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:58.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:57.945+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:58.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:57.946+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:58.617 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:58.311+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:58.617 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:58.311+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:58.617 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:58.311+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:58.617 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:58.312+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:58.986 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:58.613+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:58.986 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:58.613+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:58.986 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:58.613+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:58.986 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:58.614+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:59.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:58.982+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:59.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:58.982+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:59.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:58.982+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:15:59.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:58.983+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:59.626 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:15:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:15:59.453+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 10490 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:15:59.626 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:59.343+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:59.626 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:59.343+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:59.627 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:59.343+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:15:59.627 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:15:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:15:59.343+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:15:59.950 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:59.622+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:59.950 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:59.622+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:59.950 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:59.622+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:15:59.950 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:15:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:15:59.623+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:00.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:59.946+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:00.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:59.946+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:00.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:59.946+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:00.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:15:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:15:59.947+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:00.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:00.596+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:00.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:00.596+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:00.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:00.596+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:00.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:00.596+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:00.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:00.387+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:00.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:00.387+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:00.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:00.387+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:00.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:00.388+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:01.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:00.903+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:01.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:00.903+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:01.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:00.903+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:01.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:00.904+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:01.678 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:01.575+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:01.679 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:01.575+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:01.679 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:01.575+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:01.679 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:01.576+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:01.679 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:01.419+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:01.679 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:01.419+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:01.679 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:01.419+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:01.679 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:01.420+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:02.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:01.908+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:02.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:01.908+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:02.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:01.908+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:02.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:01.909+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:02.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:02.373+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:02.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:02.373+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:02.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:02.373+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:02.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:02.374+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:02.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:02.587+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:02.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:02.587+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:02.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:02.587+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:02.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:02.588+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:03.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:02.889+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:03.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:02.889+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:03.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:02.889+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:03.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:02.890+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:03.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:03.356+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:03.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:03.356+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:03.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:03.356+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:03.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:03.357+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:03.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:03.542+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:03.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:03.542+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:03.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:03.542+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:03.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:03.543+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:04.134 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:03.843+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:04.134 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:03.843+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:04.134 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:03.843+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:04.134 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:03.843+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:04.459 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:04.361+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:04.459 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:04.361+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:04.459 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:04.361+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:04.459 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:04.362+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:04.779 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:16:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:16:04.455+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 10584 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:16:04.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:04.589+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:04.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:04.589+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:04.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:04.589+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:04.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:04.589+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:05.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:04.891+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:05.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:04.891+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:05.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:04.891+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:05.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:04.891+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:05.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:05.578+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:05.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:05.578+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:05.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:05.578+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:05.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:05.578+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:05.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:05.391+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:05.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:05.391+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:05.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:05.391+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:05.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:05.392+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:06.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:05.930+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:06.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:05.930+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:06.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:05.930+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:06.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:05.931+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:06.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:06.606+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:06.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:06.606+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:06.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:06.606+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:06.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:06.606+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:06.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:06.394+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:06.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:06.394+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:06.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:06.394+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:06.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:06.395+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:07.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:06.962+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:07.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:06.962+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:07.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:06.962+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:07.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:06.963+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:07.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:07.384+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:07.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:07.384+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:07.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:07.384+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:07.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:07.385+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:07.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:07.610+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:07.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:07.610+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:07.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:07.610+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:07.782 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:07.611+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:08.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:07.974+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:08.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:07.974+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:08.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:07.974+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:08.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:07.975+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:08.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:08.397+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:08.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:08.397+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:08.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:08.397+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:08.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:08.398+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:08.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:08.630+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:08.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:08.630+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:08.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:08.630+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:08.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:08.631+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:09.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:08.954+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:09.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:08.954+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:09.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:08.954+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:09.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:08.954+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:09.581 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:16:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:16:09.456+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 10677 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:16:09.582 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:09.382+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:09.582 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:09.382+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:09.582 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:09.382+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:09.582 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:09.384+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:09.929 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:09.587+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:09.929 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:09.587+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:09.929 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:09.587+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:09.929 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:09.588+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:10.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:09.925+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:10.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:09.925+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:10.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:09.925+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:10.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:09.925+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:10.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:10.594+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:10.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:10.594+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:10.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:10.594+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:10.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:10.595+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:10.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:10.368+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:10.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:10.368+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:10.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:10.368+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:10.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:10.368+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:11.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:10.879+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:11.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:10.879+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:11.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:10.879+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:11.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:10.880+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:11.642 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:11.368+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:11.642 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:11.368+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:11.642 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:11.368+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:11.644 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:11.369+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:12.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:11.638+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:12.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:11.638+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:12.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:11.638+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:12.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:11.639+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:12.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:11.884+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:12.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:11.884+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:12.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:11.884+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:12.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:11.885+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:12.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:12.613+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:12.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:12.613+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:12.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:12.613+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:12.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:12.614+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:12.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:12.374+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:12.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:12.374+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:12.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:12.374+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:12.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:12.375+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:13.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:12.884+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:13.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:12.884+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:13.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:12.884+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:13.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:12.885+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:13.638 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:13.356+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:13.642 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:13.356+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:13.642 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:13.356+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:13.642 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:13.357+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:14.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:13.653+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:14.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:13.653+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:14.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:13.653+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:14.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:13.653+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:14.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:13.881+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:14.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:13.881+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:14.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:13.881+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:14.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:13.882+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:14.585 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:16:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:16:14.457+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 10770 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:16:14.586 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:14.330+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:14.586 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:14.330+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:14.586 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:14.330+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:14.586 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:14.331+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:14.838 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:14.653+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:14.838 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:14.653+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:14.838 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:14.653+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:14.838 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:14.653+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:15.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:14.834+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:15.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:14.834+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:15.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:14.834+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:15.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:14.834+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:15.618 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:15.344+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:15.619 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:15.344+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:15.619 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:15.344+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:15.619 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:15.345+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:16.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:15.833+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:16.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:15.833+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:16.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:15.833+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:16.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:15.833+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:16.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:15.613+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:16.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:15.613+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:16.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:15.613+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:16.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:15.614+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:16.656 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:16.348+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:16.656 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:16.348+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:16.657 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:16.348+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:16.657 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:16.349+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:17.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:16.652+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:17.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:16.652+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:17.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:16.652+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:17.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:16.653+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:17.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:16.848+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:17.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:16.848+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:17.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:16.848+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:17.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:16.849+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:17.700 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:17.308+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:17.700 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:17.308+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:17.700 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:17.308+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:17.701 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:17.309+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:18.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:17.696+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:18.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:17.696+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:18.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:17.696+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:18.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:17.697+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:18.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:17.867+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:18.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:17.867+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:18.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:17.867+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:18.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:17.868+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:18.684 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:18.333+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:18.685 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:18.333+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:18.685 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:18.333+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:18.685 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:18.334+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:19.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:18.871+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:19.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:18.871+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:19.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:18.871+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:19.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:18.872+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:19.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:18.680+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:19.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:18.680+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:19.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:18.680+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:19.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:18.681+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:19.585 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:16:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:16:19.458+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 10862 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:16:19.585 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:19.285+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:19.585 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:19.285+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:19.586 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:19.285+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:19.586 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:19.286+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:19.850 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:19.679+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:19.850 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:19.679+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:19.851 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:19.679+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:19.851 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:19.680+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:20.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:19.846+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:20.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:19.846+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:20.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:19.846+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:20.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:19.847+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:20.681 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:20.315+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:20.682 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:20.315+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:20.682 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:20.315+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:20.682 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:20.316+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:21.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:20.677+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:21.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:20.677+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:21.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:20.677+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:21.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:20.678+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:21.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:20.861+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:21.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:20.861+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:21.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:20.861+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:21.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:20.862+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:21.673 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:21.338+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:21.673 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:21.338+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:21.674 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:21.338+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:21.674 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:21.339+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:22.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:21.669+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:22.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:21.669+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:22.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:21.669+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:22.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:21.670+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:22.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:21.855+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:22.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:21.855+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:22.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:21.855+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:22.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:21.856+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:22.566 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:22.294+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:22.566 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:22.294+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:22.566 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:22.294+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:22.566 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:22.295+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:22.850 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:22.715+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:22.851 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:22.715+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:22.851 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:22.715+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:22.851 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:22.716+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:23.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:22.847+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:23.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:22.847+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:23.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:22.847+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:23.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:22.848+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:23.695 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:23.345+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:23.695 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:23.345+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:23.695 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:23.345+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:23.695 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:23.347+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:24.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:23.691+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:24.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:23.691+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:24.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:23.691+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:24.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:23.692+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:24.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:23.821+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:24.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:23.821+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:24.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:23.821+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:24.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:23.822+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:24.588 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:16:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:16:24.460+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 10956 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:16:24.588 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:24.316+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:24.588 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:24.316+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:24.588 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:24.316+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:24.588 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:24.317+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:25.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:24.665+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:25.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:24.665+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:25.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:24.665+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:25.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:24.666+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:25.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:24.781+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:25.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:24.781+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:25.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:24.781+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:25.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:24.782+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:25.638 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:25.324+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:25.638 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:25.324+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:25.638 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:25.324+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:25.638 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:25.325+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:26.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:25.658+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:26.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:25.658+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:26.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:25.658+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:26.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:25.659+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:26.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:25.747+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:26.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:25.747+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:26.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:25.747+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:26.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:25.749+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:26.677 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:26.316+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:26.677 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:26.316+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:26.677 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:26.316+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:26.677 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:26.317+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:27.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:26.671+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:27.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:26.671+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:27.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:26.671+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:27.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:26.672+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:27.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:26.735+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:27.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:26.735+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:27.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:26.735+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:27.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:26.736+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:27.636 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:27.333+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:27.637 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:27.333+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:27.637 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:27.333+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:27.637 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:27.333+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:28.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:27.632+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:28.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:27.632+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:28.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:27.632+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:28.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:27.633+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:28.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:27.711+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:28.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:27.711+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:28.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:27.711+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:28.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:27.712+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:28.658 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:28.366+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:28.658 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:28.366+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:28.658 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:28.366+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:28.658 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:28.367+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:29.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:28.654+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:29.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:28.654+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:29.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:28.654+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:29.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:28.655+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:29.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:28.761+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:29.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:28.761+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:29.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:28.761+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:29.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:28.761+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:29.658 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:16:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:16:29.461+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 11050 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:16:29.659 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:29.408+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:29.659 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:29.408+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:29.659 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:29.408+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:29.659 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:29.409+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:30.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:29.654+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:30.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:29.654+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:30.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:29.654+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:30.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:29.656+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:30.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:29.717+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:30.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:29.717+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:30.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:29.717+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:30.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:29.718+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:30.745 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:30.624+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:30.745 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:30.624+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:30.745 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:30.624+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:30.745 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:30.624+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:30.745 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:30.383+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:30.746 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:30.383+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:30.746 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:30.383+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:30.746 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:30.384+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:31.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:30.741+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:31.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:30.741+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:31.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:30.741+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:31.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:30.742+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:31.674 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:31.407+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:31.674 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:31.407+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:31.674 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:31.407+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:31.674 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:31.407+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:32.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:31.667+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:32.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:31.667+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:32.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:31.667+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:32.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:31.668+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:32.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:31.758+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:32.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:31.758+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:32.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:31.758+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:32.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:31.758+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:32.718 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:32.372+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:32.719 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:32.372+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:32.719 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:32.372+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:32.719 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:32.372+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:33.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:32.743+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:33.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:32.743+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:33.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:32.743+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:33.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:32.744+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:33.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:32.715+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:33.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:32.715+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:33.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:32.715+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:33.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:32.715+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:33.740 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:33.421+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:33.740 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:33.421+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:33.740 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:33.421+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:33.740 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:33.421+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:34.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:33.736+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:34.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:33.736+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:34.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:33.736+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:34.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:33.737+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:34.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:33.740+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:34.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:33.740+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:34.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:33.740+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:34.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:33.740+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:34.745 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:16:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:16:34.462+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 11143 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:16:34.746 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:34.460+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:34.746 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:34.460+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:34.746 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:34.460+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:34.746 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:34.460+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:34.746 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:34.703+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:34.746 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:34.703+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:34.746 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:34.703+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:34.746 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:34.704+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:35.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:34.739+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:35.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:34.739+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:35.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:34.739+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:35.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:34.740+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:35.742 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:35.718+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:35.742 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:35.718+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:35.743 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:35.718+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:35.743 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:35.718+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:35.743 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:35.486+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:35.743 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:35.486+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:35.743 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:35.486+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:35.743 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:35.487+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:36.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:35.738+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:36.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:35.738+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:36.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:35.738+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:36.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:35.739+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:36.744 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:36.473+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:36.744 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:36.473+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:36.744 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:36.473+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:36.744 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:36.474+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:37.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:36.757+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:37.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:36.757+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:37.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:36.757+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:37.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:36.757+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:37.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:36.739+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:37.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:36.739+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:37.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:36.739+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:37.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:36.740+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:37.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:37.491+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:37.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:37.491+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:37.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:37.491+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:37.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:37.492+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:37.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:37.729+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:37.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:37.729+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:37.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:37.729+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:37.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:37.730+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:38.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:37.800+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:38.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:37.800+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:38.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:37.800+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:38.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:37.801+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:38.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:38.506+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:38.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:38.506+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:38.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:38.506+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:38.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:38.507+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:38.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:38.752+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:38.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:38.752+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:38.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:38.752+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:38.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:38.753+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:39.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:38.839+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:39.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:38.839+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:39.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:38.839+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:39.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:38.840+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:39.593 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:16:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:16:39.464+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 11236 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:16:39.593 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:39.499+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:39.593 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:39.499+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:39.593 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:39.499+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:39.593 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:39.500+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:40.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:39.740+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:40.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:39.740+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:40.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:39.740+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:40.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:39.740+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:40.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:39.798+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:40.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:39.798+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:40.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:39.798+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:40.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:39.799+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:40.761 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:40.457+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:40.761 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:40.457+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:40.761 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:40.457+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:40.761 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:40.457+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:41.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:40.758+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:41.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:40.758+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:41.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:40.758+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:41.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:40.759+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:41.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:40.757+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:41.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:40.757+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:41.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:40.757+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:41.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:40.757+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:41.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:41.715+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:41.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:41.715+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:41.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:41.715+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:41.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:41.715+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:41.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:41.479+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:41.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:41.479+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:41.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:41.479+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:41.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:41.480+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:41.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:41.714+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:41.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:41.714+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:41.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:41.714+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:41.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:41.715+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:42.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:42.704+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:42.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:42.704+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:42.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:42.704+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:42.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:42.705+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:42.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:42.478+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:42.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:42.478+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:42.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:42.478+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:42.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:42.479+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:42.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:42.698+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:42.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:42.698+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:42.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:42.698+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:42.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:42.698+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:43.737 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:43.440+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:43.737 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:43.440+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:43.737 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:43.440+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:43.737 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:43.441+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:44.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:43.737+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:44.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:43.737+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:44.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:43.737+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:44.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:43.738+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:44.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:43.733+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:44.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:43.733+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:44.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:43.733+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:44.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:43.734+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:44.702 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:16:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:16:44.465+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 11330 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:16:44.703 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:44.440+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:44.703 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:44.440+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:44.703 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:44.440+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:44.703 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:44.441+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:45.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:44.698+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:45.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:44.698+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:45.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:44.698+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:45.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:44.699+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:45.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:44.717+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:45.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:44.717+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:45.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:44.717+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:45.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:44.718+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:45.706 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:45.408+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:45.706 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:45.408+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:45.706 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:45.408+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:45.706 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:45.409+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:45.707 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:45.656+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:45.707 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:45.656+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:45.707 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:45.656+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:45.707 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:45.657+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:46.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:45.702+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:46.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:45.702+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:46.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:45.702+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:46.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:45.703+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:46.724 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:46.608+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:46.724 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:46.608+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:46.724 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:46.608+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:46.724 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:46.609+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:46.724 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:46.409+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:46.724 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:46.409+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:46.724 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:46.409+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:46.724 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:46.409+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:47.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:46.719+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:47.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:46.719+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:47.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:46.719+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:47.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:46.720+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:47.727 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:47.560+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:47.728 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:47.560+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:47.728 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:47.560+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:47.728 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:47.560+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:47.728 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:47.455+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:47.728 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:47.455+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:47.728 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:47.455+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:47.728 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:47.455+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:48.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:47.723+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:48.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:47.723+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:48.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:47.723+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:48.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:47.724+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:48.748 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:48.554+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:48.749 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:48.554+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:48.749 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:48.554+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:48.749 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:48.555+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:48.749 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:48.414+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:48.749 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:48.414+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:48.749 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:48.414+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:48.749 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:48.415+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:49.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:48.744+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:49.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:48.744+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:49.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:48.744+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:49.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:48.745+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:49.750 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:16:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:16:49.466+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 11424 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:16:49.750 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:49.398+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:49.750 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:49.398+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:49.751 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:49.398+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:49.751 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:49.399+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:49.751 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:49.531+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:49.751 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:49.531+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:49.751 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:49.531+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:49.751 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:49.532+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:50.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:49.746+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:50.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:49.746+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:50.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:49.746+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:50.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:49.747+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:50.779 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:50.491+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:50.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:50.491+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:50.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:50.491+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:50.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:50.492+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:50.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:50.370+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:50.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:50.370+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:50.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:50.370+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:50.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:50.371+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:51.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:50.775+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:51.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:50.775+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:51.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:50.775+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:51.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:50.776+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:51.682 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:51.338+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:51.682 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:51.338+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:51.682 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:51.338+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:51.682 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:51.338+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:51.682 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:51.489+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:51.683 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:51.489+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:51.683 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:51.489+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:51.683 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:51.490+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:52.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:51.767+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:52.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:51.767+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:52.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:51.767+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:52.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:51.768+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:52.726 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:52.307+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:52.726 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:52.307+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:52.726 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:52.307+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:52.726 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:52.307+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:52.727 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:52.456+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:52.727 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:52.456+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:52.727 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:52.456+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:52.727 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:52.457+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:53.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:52.722+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:53.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:52.722+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:53.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:52.722+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:53.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:52.723+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:53.568 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:53.282+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:53.568 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:53.282+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:53.568 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:53.282+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:53.568 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:53.283+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:53.569 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:53.413+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:53.569 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:53.413+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:53.569 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:53.413+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:53.569 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:53.414+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:54.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:53.740+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:54.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:53.740+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:54.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:53.740+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:54.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:53.741+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:54.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:16:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:16:54.468+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 11517 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:16:54.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:54.252+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:54.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:54.252+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:54.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:54.252+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:54.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:54.253+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:54.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:54.415+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:54.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:54.415+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:54.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:54.415+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:54.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:54.416+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:55.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:54.760+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:55.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:54.760+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:55.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:54.760+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:55.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:54.761+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:55.739 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:55.300+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:55.740 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:55.300+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:55.740 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:55.300+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:55.740 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:55.301+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:55.740 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:55.417+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:55.740 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:55.417+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:55.740 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:55.417+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:55.740 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:55.418+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:56.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:55.735+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:56.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:55.735+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:56.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:55.735+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:56.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:55.736+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:56.690 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:56.368+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:56.691 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:56.368+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:56.691 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:56.368+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:56.691 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:56.369+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:56.691 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:56.298+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:56.691 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:56.298+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:56.691 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:56.298+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:56.691 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:56.299+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:57.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:56.686+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:57.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:56.686+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:57.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:56.686+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:57.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:56.687+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:57.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:57.273+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:57.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:57.273+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:57.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:57.273+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:57.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:57.274+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:57.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:57.354+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:57.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:57.354+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:57.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:57.354+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:57.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:57.355+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:58.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:57.686+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:58.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:57.686+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:58.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:57.686+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:58.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:57.687+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:58.689 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:58.319+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:58.689 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:58.319+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:58.689 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:58.319+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:58.689 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:58.320+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:58.689 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:58.311+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:58.689 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:58.311+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:58.689 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:58.311+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:58.689 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:58.312+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:59.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:58.684+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:59.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:58.684+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:59.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:58.684+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:16:59.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:58.685+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:59.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:16:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:16:59.469+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 11610 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:16:59.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:59.356+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:59.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:59.356+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:59.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:59.356+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:16:59.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:16:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:16:59.357+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:16:59.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:59.275+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:59.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:59.275+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:59.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:59.275+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:16:59.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:16:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:16:59.276+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:00.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:59.658+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:00.075 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:59.658+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:00.075 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:59.658+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:00.076 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:16:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:16:59.659+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:00.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:00.388+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:00.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:00.388+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:00.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:00.388+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:00.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:00.388+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:00.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:00.254+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:00.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:00.254+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:00.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:00.254+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:00.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:00.255+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:01.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:00.661+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:01.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:00.661+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:01.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:00.661+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:01.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:00.662+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:01.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:01.266+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:01.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:01.266+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:01.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:01.266+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:01.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:01.267+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:01.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:01.346+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:01.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:01.346+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:01.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:01.346+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:01.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:01.347+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:02.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:01.630+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:02.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:01.630+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:02.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:01.630+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:02.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:01.631+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:02.643 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:02.375+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:02.643 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:02.375+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:02.643 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:02.375+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:02.643 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:02.376+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:02.644 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:02.285+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:02.644 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:02.285+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:02.644 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:02.285+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:02.644 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:02.286+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:03.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:02.638+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:03.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:02.638+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:03.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:02.638+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:03.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:02.639+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:03.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:03.329+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:03.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:03.329+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:03.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:03.329+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:03.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:03.331+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:03.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:03.266+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:03.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:03.266+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:03.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:03.266+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:03.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:03.267+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:04.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:03.677+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:04.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:03.677+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:04.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:03.677+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:04.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:03.678+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:04.599 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:17:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:17:04.471+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 11704 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:17:04.599 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:04.326+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:04.600 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:04.326+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:04.600 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:04.326+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:04.600 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:04.327+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:04.600 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:04.296+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:04.600 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:04.296+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:04.600 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:04.296+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:04.600 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:04.297+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:05.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:04.647+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:05.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:04.647+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:05.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:04.647+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:05.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:04.648+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:05.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:05.309+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:05.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:05.309+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:05.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:05.309+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:05.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:05.309+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:05.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:05.250+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:05.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:05.250+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:05.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:05.250+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:05.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:05.251+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:06.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:05.687+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:06.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:05.687+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:06.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:05.687+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:06.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:05.688+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:06.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:06.352+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:06.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:06.352+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:06.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:06.352+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:06.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:06.353+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:06.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:06.211+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:06.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:06.211+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:06.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:06.211+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:06.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:06.212+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:07.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:06.717+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:07.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:06.717+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:07.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:06.717+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:07.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:06.718+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:07.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:07.365+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:07.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:07.365+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:07.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:07.365+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:07.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:07.365+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:07.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:07.195+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:07.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:07.195+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:07.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:07.195+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:07.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:07.195+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:08.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:07.746+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:08.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:07.746+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:08.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:07.746+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:08.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:07.747+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:08.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:08.383+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:08.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:08.383+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:08.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:08.383+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:08.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:08.384+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:08.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:08.196+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:08.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:08.196+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:08.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:08.196+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:08.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:08.197+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:09.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:08.772+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:09.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:08.772+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:09.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:08.772+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:09.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:08.773+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:09.388 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:09.229+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:09.388 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:09.229+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:09.388 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:09.229+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:09.388 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:09.230+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:09.745 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:09.411+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:09.745 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:09.411+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:09.745 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:09.411+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:09.745 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:09.412+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:09.746 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:17:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:17:09.472+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 11797 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:17:10.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:09.741+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:10.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:09.741+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:10.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:09.741+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:10.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:09.742+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:10.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:10.406+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:10.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:10.407+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:10.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:10.407+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:10.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:10.408+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:10.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:10.216+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:10.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:10.216+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:10.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:10.216+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:10.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:10.216+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:11.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:10.736+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:11.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:10.736+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:11.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:10.736+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:11.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:10.736+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:11.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:11.374+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:11.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:11.374+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:11.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:11.374+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:11.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:11.375+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:11.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:11.258+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:11.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:11.258+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:11.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:11.258+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:11.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:11.259+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:12.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:11.784+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:12.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:11.784+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:12.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:11.784+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:12.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:11.785+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:12.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:12.361+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:12.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:12.361+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:12.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:12.361+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:12.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:12.362+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:12.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:12.264+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:12.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:12.264+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:12.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:12.264+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:12.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:12.264+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:13.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:12.780+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:13.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:12.780+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:13.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:12.780+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:13.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:12.780+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:13.570 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:13.284+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:13.570 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:13.284+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:13.570 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:13.284+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:13.570 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:13.285+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:13.570 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:13.372+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:13.571 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:13.372+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:13.571 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:13.372+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:13.571 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:13.372+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:14.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:13.787+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:14.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:13.787+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:14.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:13.787+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:14.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:13.788+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:14.501 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:17:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:17:14.474+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 11889 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:17:14.502 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:14.362+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:14.502 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:14.362+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:14.502 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:14.362+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:14.502 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:14.362+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:14.502 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:14.240+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:14.502 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:14.240+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:14.502 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:14.240+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:14.503 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:14.241+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:15.206 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:14.787+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:15.206 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:14.787+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:15.206 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:14.787+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:15.206 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:14.787+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:15.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:15.202+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:15.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:15.202+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:15.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:15.202+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:15.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:15.203+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:15.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:15.319+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:15.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:15.319+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:15.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:15.319+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:15.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:15.320+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:16.232 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:15.809+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:16.233 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:15.809+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:16.233 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:15.809+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:16.233 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:15.809+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:16.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:16.338+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:16.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:16.338+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:16.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:16.338+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:16.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:16.339+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:16.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:16.228+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:16.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:16.228+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:16.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:16.228+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:16.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:16.229+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:17.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:16.803+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:17.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:16.803+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:17.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:16.803+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:17.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:16.804+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:17.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:17.358+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:17.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:17.358+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:17.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:17.358+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:17.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:17.359+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:17.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:17.272+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:17.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:17.272+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:17.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:17.272+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:17.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:17.273+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:18.248 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:17.812+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:18.248 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:17.812+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:18.248 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:17.812+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:18.248 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:17.813+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:18.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:18.244+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:18.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:18.244+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:18.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:18.244+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:18.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:18.245+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:18.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:18.374+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:18.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:18.374+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:18.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:18.374+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:18.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:18.375+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:19.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:18.785+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:19.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:18.785+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:19.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:18.785+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:19.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:18.786+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:19.604 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:17:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:17:19.475+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 11981 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:17:19.604 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:19.294+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:19.604 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:19.294+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:19.604 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:19.294+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:19.604 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:19.295+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:19.604 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:19.346+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:19.605 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:19.346+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:19.605 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:19.346+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:19.605 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:19.347+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:20.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:19.740+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:20.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:19.740+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:20.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:19.740+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:20.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:19.741+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:20.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:20.347+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:20.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:20.347+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:20.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:20.347+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:20.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:20.348+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:20.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:20.255+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:20.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:20.255+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:20.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:20.255+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:20.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:20.256+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:21.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:20.786+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:21.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:20.786+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:21.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:20.786+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:21.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:20.786+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:21.684 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:21.295+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:21.684 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:21.295+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:21.684 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:21.295+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:21.684 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:21.296+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:21.684 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:21.367+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:21.684 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:21.367+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:21.685 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:21.367+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:21.685 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:21.368+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:22.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:21.769+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:22.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:21.769+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:22.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:21.769+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:22.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:21.769+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:22.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:22.335+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:22.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:22.335+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:22.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:22.335+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:22.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:22.336+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:22.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:22.315+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:22.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:22.315+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:22.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:22.315+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:22.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:22.316+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:23.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:22.780+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:23.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:22.780+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:23.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:22.780+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:23.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:22.781+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:23.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:23.360+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:23.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:23.360+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:23.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:23.360+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:23.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:23.360+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:23.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:23.359+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:23.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:23.359+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:23.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:23.359+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:23.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:23.360+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:24.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:23.812+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:24.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:23.812+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:24.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:23.812+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:24.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:23.812+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:24.604 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:17:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:17:24.476+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 12074 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:17:24.605 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:24.323+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:24.605 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:24.323+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:24.605 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:24.323+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:24.605 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:24.323+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:24.605 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:24.356+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:24.605 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:24.356+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:24.605 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:24.356+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:24.605 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:24.357+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:25.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:24.778+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:25.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:24.778+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:25.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:24.778+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:25.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:24.778+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:25.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:25.360+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:25.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:25.360+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:25.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:25.360+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:25.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:25.360+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:25.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:25.326+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:25.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:25.326+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:25.532 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:25.326+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:25.532 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:25.326+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:26.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:25.790+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:26.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:25.790+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:26.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:25.790+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:26.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:25.791+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:26.747 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:26.392+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:26.747 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:26.392+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:26.747 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:26.392+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:26.747 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:26.393+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:26.748 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:26.361+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:26.748 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:26.361+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:26.748 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:26.361+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:26.748 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:26.362+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:27.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:26.743+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:27.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:26.743+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:27.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:26.743+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:27.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:26.744+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:27.718 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:27.386+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:27.719 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:27.386+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:27.719 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:27.386+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:27.719 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:27.387+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:27.719 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:27.346+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:27.719 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:27.346+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:27.719 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:27.346+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:27.719 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:27.348+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:28.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:27.714+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:28.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:27.714+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:28.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:27.714+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:28.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:27.715+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:28.676 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:28.343+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:28.676 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:28.343+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:28.676 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:28.343+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:28.676 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:28.344+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:28.676 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:28.357+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:28.676 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:28.357+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:28.676 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:28.357+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:28.676 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:28.358+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:29.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:28.672+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:29.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:28.672+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:29.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:28.672+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:29.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:28.672+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:29.605 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:17:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:17:29.478+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 12167 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:17:29.606 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:29.329+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:29.606 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:29.329+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:29.606 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:29.329+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:29.606 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:29.330+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:29.606 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:29.310+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:29.606 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:29.310+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:29.606 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:29.310+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:29.607 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:29.311+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:30.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:29.628+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:30.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:29.628+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:30.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:29.628+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:30.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:29.628+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:30.587 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:30.373+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:30.614 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:30.373+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:30.615 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:30.373+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:30.615 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:30.374+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:30.615 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:30.298+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:30.615 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:30.298+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:30.615 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:30.298+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:30.615 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:30.298+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:31.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:30.582+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:31.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:30.582+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:31.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:30.582+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:31.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:30.583+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:31.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:31.271+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:31.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:31.271+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:31.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:31.271+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:31.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:31.272+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:31.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:31.360+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:31.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:31.360+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:31.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:31.360+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:31.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:31.360+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:32.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:31.562+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:32.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:31.562+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:32.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:31.562+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:32.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:31.563+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:32.566 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:32.407+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:32.567 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:32.407+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:32.567 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:32.407+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:32.567 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:32.408+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:32.567 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:32.296+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:32.567 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:32.296+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:32.567 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:32.296+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:32.567 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:32.297+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:33.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:32.562+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:33.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:32.562+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:33.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:32.562+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:33.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:32.563+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:33.520 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:33.425+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:33.520 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:33.425+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:33.520 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:33.425+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:33.520 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:33.425+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:33.520 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:33.333+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:33.521 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:33.333+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:33.521 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:33.333+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:33.521 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:33.334+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:33.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:33.516+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:33.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:33.516+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:33.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:33.516+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:33.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:33.517+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:34.608 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:17:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:17:34.479+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 12259 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:17:34.608 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:34.438+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:34.608 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:34.438+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:34.609 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:34.438+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:34.609 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:34.439+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:34.609 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:34.532+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:34.609 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:34.532+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:34.609 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:34.532+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:34.609 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:34.532+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:34.609 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:34.329+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:34.609 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:34.329+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:34.609 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:34.329+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:34.610 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:34.330+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:35.580 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:35.439+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:35.580 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:35.439+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:35.580 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:35.439+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:35.580 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:35.440+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:35.580 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:35.319+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:35.580 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:35.319+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:35.580 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:35.319+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:35.581 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:35.320+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:36.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:35.576+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:36.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:35.576+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:36.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:35.576+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:36.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:35.576+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:36.589 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:36.420+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:36.589 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:36.420+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:36.589 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:36.420+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:36.590 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:36.421+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:36.590 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:36.307+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:36.590 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:36.307+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:36.590 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:36.307+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:36.590 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:36.308+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:37.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:36.585+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:37.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:36.585+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:37.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:36.585+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:37.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:36.586+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:37.570 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:37.436+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:37.571 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:37.436+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:37.571 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:37.436+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:37.571 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:37.436+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:37.571 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:37.295+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:37.571 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:37.295+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:37.571 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:37.295+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:37.571 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:37.296+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:37.912 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:37.566+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:37.912 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:37.566+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:37.912 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:37.566+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:37.912 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:37.567+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:37.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-03-11T23:17:37.907+0000 7fdf6184c700 0 monclient(hunting): authenticate timed out after 300 2024-03-11T23:17:37.913 INFO:teuthology.orchestra.run.smithi153.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-03-11T23:17:38.560 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:38.455+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:38.560 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:38.455+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:38.560 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:38.455+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:38.560 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:38.456+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:38.560 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:38.316+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:38.560 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:38.316+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:38.561 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:38.316+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:38.561 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:38.317+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:38.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T23:17:38.734 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/cephadm.py", line 1364, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/cephadm.py", line 39, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi153 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2024-03-11T23:17:39.063 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=91c4299b4a3b4e8fbda1931f0337924d Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/cephadm.py", line 1364, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/cephadm.py", line 39, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi153 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2024-03-11T23:17:39.066 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-11T23:17:39.080 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-11T23:17:39.080 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-11T23:17:39.080 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-11T23:17:39.085 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:38.556+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:39.086 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:38.556+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:39.086 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:38.556+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:39.086 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:38.556+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:39.105 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-11T23:17:39.106 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-11T23:17:39.162 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T23:17:39.591 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:17:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:17:39.480+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 12350 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:17:39.592 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:39.409+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:39.592 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:39.409+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:39.592 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:39.409+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:39.592 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:39.410+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:39.593 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:39.280+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:39.593 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:39.280+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:39.593 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:39.280+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:39.593 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:39.281+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:40.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:39.587+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:40.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:39.587+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:40.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:39.587+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:40.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:39.588+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:40.591 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:40.459+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:40.592 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:40.459+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:40.592 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:40.459+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:40.592 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:40.459+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:40.592 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:40.319+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:40.592 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:40.319+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:40.592 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:40.319+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:40.593 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:40.320+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:40.904 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:40.587+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:40.904 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:40.587+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:40.904 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:40.587+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:40.904 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:40.589+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:41.603 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:41.418+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:41.603 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:41.418+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:41.603 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:41.418+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:41.603 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:41.419+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:41.603 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:41.323+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:41.604 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:41.323+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:41.604 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:41.323+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:41.604 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:41.324+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:42.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:41.599+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:42.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:41.599+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:42.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:41.599+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:42.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:41.599+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:42.618 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:42.375+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:42.619 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:42.375+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:42.619 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:42.375+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:42.619 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:42.376+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:42.619 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:42.308+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:42.620 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:42.308+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:42.620 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:42.308+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:42.620 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:42.309+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:43.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:42.614+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:43.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:42.614+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:43.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:42.614+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:43.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:42.616+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:43.572 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:43.345+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:43.573 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:43.345+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:43.573 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:43.345+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:43.573 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:43.345+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:43.573 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:43.285+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:43.573 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:43.285+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:43.573 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:43.285+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:43.573 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:43.285+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:44.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:43.593+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:44.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:43.593+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:44.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:43.593+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:44.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:43.594+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:44.600 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:17:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:17:44.482+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 12443 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:17:44.601 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:44.360+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:44.601 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:44.360+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:44.601 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:44.360+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:44.601 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:44.361+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:44.602 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:44.307+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:44.602 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:44.307+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:44.602 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:44.307+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:44.602 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:44.308+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:45.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:44.595+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:45.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:44.595+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:45.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:44.595+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:45.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:44.596+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:45.609 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:45.315+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:45.609 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:45.315+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:45.610 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:45.315+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:45.610 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:45.316+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:45.610 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:45.316+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:45.610 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:45.316+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:45.610 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:45.316+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:45.611 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:45.316+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:46.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:45.605+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:46.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:45.605+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:46.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:45.605+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:46.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:45.605+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:46.621 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:46.290+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:46.621 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:46.290+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:46.622 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:46.290+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:46.622 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:46.291+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:46.622 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:46.313+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:46.622 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:46.313+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:46.622 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:46.313+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:46.622 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:46.314+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:47.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:46.617+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:47.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:46.617+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:47.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:46.617+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:47.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:46.617+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:47.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:47.266+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:47.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:47.266+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:47.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:47.266+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:47.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:47.267+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:47.532 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:47.293+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:47.532 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:47.293+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:47.532 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:47.293+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:47.532 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:47.294+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:48.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:47.570+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:48.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:47.570+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:48.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:47.570+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:48.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:47.570+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:48.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:48.268+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:48.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:48.268+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:48.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:48.268+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:48.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:48.269+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:48.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:48.231+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:48.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:48.231+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:48.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:48.231+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:48.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:48.232+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:49.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:48.598+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:49.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:48.598+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:49.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:48.598+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:49.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:48.599+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:49.488 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:49.225+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:49.488 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:49.225+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:49.488 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:49.225+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:49.489 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:49.226+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:49.489 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:49.269+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:49.489 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:49.269+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:49.489 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:49.269+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:49.490 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:49.270+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:49.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:17:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:17:49.483+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 12536 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:17:49.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:49.627+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:49.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:49.627+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:49.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:49.627+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:49.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:49.628+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:50.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:50.227+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:50.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:50.227+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:50.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:50.227+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:50.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:50.228+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:50.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:50.292+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:50.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:50.292+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:50.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:50.292+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:50.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:50.293+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:51.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:50.591+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:51.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:50.591+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:51.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:50.591+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:51.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:50.592+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:51.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:51.291+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:51.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:51.291+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:51.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:51.291+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:51.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:51.291+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:51.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:51.227+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:51.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:51.227+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:51.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:51.227+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:51.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:51.228+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:52.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:51.616+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:52.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:51.616+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:52.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:51.616+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:52.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:51.616+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:52.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:52.262+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:52.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:52.262+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:52.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:52.262+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:52.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:52.264+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:52.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:52.260+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:52.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:52.260+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:52.532 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:52.260+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:52.532 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:52.261+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:53.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:52.574+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:53.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:52.574+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:53.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:52.574+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:53.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:52.575+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:53.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:53.293+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:53.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:53.293+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:53.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:53.293+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:53.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:53.294+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:53.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:53.224+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:53.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:53.224+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:53.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:53.224+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:53.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:53.224+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:54.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:53.535+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:54.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:53.535+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:54.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:53.535+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:54.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:53.536+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:54.489 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:54.230+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:54.489 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:54.230+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:54.489 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:54.230+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:54.489 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:54.231+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:54.490 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:54.292+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:54.490 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:54.292+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:54.490 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:54.292+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:54.490 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:54.293+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:54.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:17:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:17:54.485+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 12628 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:17:54.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:54.531+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:54.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:54.531+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:54.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:54.531+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:54.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:54.532+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:55.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:55.224+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:55.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:55.224+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:55.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:55.224+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:55.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:55.225+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:55.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:55.282+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:55.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:55.282+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:55.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:55.282+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:55.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:55.283+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:56.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:55.527+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:56.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:55.527+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:56.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:55.527+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:56.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:55.527+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:56.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:56.196+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:56.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:56.196+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:56.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:56.196+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:56.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:56.197+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:56.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:56.279+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:56.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:56.279+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:56.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:56.279+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:56.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:56.280+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:57.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:56.535+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:57.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:56.535+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:57.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:56.535+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:57.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:56.536+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:57.293 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:57.218+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:57.293 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:57.218+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:57.293 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:57.218+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:57.293 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:57.219+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:57.293 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:57.235+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:57.294 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:57.235+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:57.294 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:57.235+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:57.294 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:57.236+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:58.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:57.546+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:58.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:57.546+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:58.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:57.546+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:58.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:57.548+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:58.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:58.252+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:58.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:58.252+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:58.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:58.252+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:58.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:58.252+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:58.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:58.191+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:58.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:58.191+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:58.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:58.191+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:58.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:58.192+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:59.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:58.581+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:59.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:58.581+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:59.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:58.581+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:59.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:58.582+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:59.490 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:59.201+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:59.490 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:59.201+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:59.491 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:59.201+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:17:59.491 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:17:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:17:59.202+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:59.491 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:59.210+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:59.491 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:59.210+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:59.491 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:59.210+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:17:59.491 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:17:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:17:59.211+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:17:59.779 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:17:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:17:59.486+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 12721 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:17:59.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:59.588+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:59.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:59.588+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:59.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:59.588+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:17:59.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:17:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:17:59.589+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:00.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:00.240+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:00.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:00.240+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:00.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:00.240+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:00.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:00.241+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:00.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:00.168+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:00.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:00.168+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:00.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:00.168+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:00.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:00.169+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:01.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:00.572+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:01.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:00.572+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:01.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:00.572+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:01.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:00.573+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:01.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:01.223+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:01.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:01.223+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:01.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:01.223+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:01.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:01.224+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:01.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:01.186+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:01.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:01.186+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:01.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:01.186+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:01.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:01.186+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:01.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:01.524+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:01.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:01.524+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:01.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:01.524+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:01.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:01.525+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:02.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:02.203+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:02.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:02.203+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:02.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:02.203+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:02.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:02.204+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:02.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:02.269+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:02.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:02.269+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:02.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:02.269+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:02.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:02.270+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:03.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:02.571+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:03.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:02.571+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:03.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:02.571+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:03.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:02.572+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:03.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:03.284+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:03.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:03.284+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:03.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:03.284+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:03.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:03.284+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:03.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:03.227+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:03.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:03.227+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:03.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:03.227+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:03.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:03.227+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:03.943 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:03.580+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:03.943 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:03.580+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:03.943 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:03.580+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:03.943 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:03.581+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:04.515 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:04.292+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:04.515 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:04.292+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:04.515 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:04.292+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:04.515 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:04.293+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:04.516 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:04.255+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:04.516 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:04.255+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:04.516 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:04.255+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:04.516 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:04.256+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:04.516 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:18:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:18:04.487+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 12815 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:18:04.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:04.587+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:04.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:04.587+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:04.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:04.587+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:04.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:04.588+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:05.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:05.290+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:05.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:05.290+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:05.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:05.290+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:05.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:05.290+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:05.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:05.261+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:05.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:05.261+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:05.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:05.261+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:05.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:05.262+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:06.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:05.627+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:06.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:05.627+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:06.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:05.627+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:06.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:05.628+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:06.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:06.222+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:06.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:06.222+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:06.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:06.222+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:06.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:06.223+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:06.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:06.248+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:06.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:06.248+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:06.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:06.248+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:06.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:06.248+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:07.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:06.675+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:07.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:06.675+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:07.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:06.675+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:07.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:06.677+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:07.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:07.249+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:07.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:07.249+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:07.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:07.249+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:07.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:07.250+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:07.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:07.251+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:07.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:07.251+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:07.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:07.251+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:07.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:07.252+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:08.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:07.689+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:08.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:07.689+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:08.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:07.689+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:08.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:07.689+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:08.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:08.222+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:08.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:08.222+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:08.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:08.222+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:08.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:08.223+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:08.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:08.237+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:08.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:08.237+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:08.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:08.237+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:08.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:08.238+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:09.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:08.709+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:09.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:08.709+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:09.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:08.709+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:09.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:08.710+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:09.388 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:09.262+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:09.388 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:09.262+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:09.389 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:09.262+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:09.389 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:09.263+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:09.389 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:09.220+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:09.389 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:09.220+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:09.389 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:09.220+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:09.389 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:09.221+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:09.734 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:18:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:18:09.489+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 12909 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:18:10.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:09.730+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:10.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:09.730+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:10.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:09.730+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:10.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:09.731+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:10.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:10.186+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:10.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:10.186+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:10.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:10.186+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:10.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:10.187+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:10.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:10.283+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:10.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:10.283+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:10.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:10.283+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:10.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:10.284+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:11.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:10.690+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:11.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:10.690+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:11.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:10.690+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:11.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:10.691+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:11.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:11.280+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:11.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:11.280+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:11.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:11.280+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:11.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:11.281+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:11.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:11.176+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:11.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:11.176+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:11.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:11.176+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:11.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:11.177+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:12.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:11.643+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:12.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:11.643+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:12.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:11.643+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:12.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:11.644+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:12.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:12.154+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:12.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:12.154+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:12.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:12.154+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:12.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:12.155+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:12.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:12.238+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:12.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:12.238+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:12.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:12.238+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:12.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:12.239+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:13.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:12.657+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:13.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:12.657+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:13.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:12.657+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:13.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:12.658+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:13.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:13.172+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:13.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:13.172+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:13.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:13.172+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:13.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:13.173+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:13.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:13.244+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:13.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:13.244+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:13.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:13.244+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:13.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:13.244+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:14.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:13.665+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:14.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:13.665+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:14.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:13.665+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:14.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:13.666+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:14.495 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:14.233+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:14.495 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:14.233+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:14.495 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:14.233+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:14.495 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:14.234+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:14.496 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:14.188+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:14.496 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:14.188+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:14.496 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:14.188+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:14.496 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:14.189+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:14.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:18:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:18:14.490+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 13002 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:18:14.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:14.649+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:14.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:14.649+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:14.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:14.649+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:14.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:14.650+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:15.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:15.197+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:15.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:15.197+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:15.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:15.197+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:15.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:15.197+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:15.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:15.227+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:15.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:15.227+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:15.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:15.227+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:15.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:15.228+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:16.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:15.648+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:16.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:15.648+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:16.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:15.648+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:16.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:15.649+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:16.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:16.272+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:16.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:16.272+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:16.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:16.272+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:16.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:16.273+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:16.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:16.148+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:16.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:16.148+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:16.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:16.148+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:16.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:16.148+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:17.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:16.674+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:17.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:16.675+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:17.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:16.675+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:17.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:16.675+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:17.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:17.161+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:17.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:17.161+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:17.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:17.161+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:17.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:17.162+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:17.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:17.318+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:17.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:17.318+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:17.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:17.318+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:17.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:17.319+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:18.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:17.722+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:18.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:17.722+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:18.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:17.722+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:18.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:17.723+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:18.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:18.303+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:18.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:18.303+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:18.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:18.303+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:18.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:18.304+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:18.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:18.182+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:18.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:18.182+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:18.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:18.182+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:18.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:18.183+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:19.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:18.712+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:19.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:18.712+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:19.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:18.712+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:19.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:18.713+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:19.496 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:19.138+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:19.497 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:19.138+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:19.497 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:19.138+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:19.497 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:19.138+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:19.497 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:19.320+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:19.497 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:19.320+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:19.497 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:19.320+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:19.498 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:19.321+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:19.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:18:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:18:19.492+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 13096 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:18:19.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:19.666+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:19.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:19.666+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:19.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:19.666+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:19.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:19.667+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:20.372 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:20.116+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:20.373 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:20.116+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:20.373 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:20.116+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:20.373 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:20.117+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:20.673 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:20.368+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:20.673 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:20.368+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:20.673 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:20.368+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:20.674 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:20.369+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:21.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:20.669+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:21.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:20.669+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:21.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:20.669+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:21.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:20.670+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:21.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:21.330+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:21.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:21.330+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:21.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:21.330+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:21.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:21.331+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:21.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:21.155+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:21.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:21.155+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:21.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:21.155+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:21.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:21.155+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:22.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:21.636+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:22.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:21.636+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:22.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:21.636+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:22.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:21.637+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:22.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:22.127+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:22.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:22.127+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:22.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:22.127+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:22.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:22.128+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:22.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:22.356+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:22.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:22.356+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:22.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:22.356+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:22.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:22.357+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:23.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:22.649+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:23.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:22.649+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:23.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:22.649+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:23.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:22.649+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:23.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:23.311+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:23.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:23.311+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:23.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:23.311+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:23.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:23.313+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:23.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:23.119+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:23.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:23.119+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:23.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:23.119+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:23.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:23.120+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:24.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:23.677+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:24.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:23.677+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:24.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:23.677+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:24.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:23.678+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:24.498 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:24.073+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:24.498 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:24.073+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:24.498 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:24.073+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:24.498 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:24.075+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:24.498 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:24.302+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:24.499 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:24.302+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:24.499 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:24.302+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:24.499 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:24.303+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:24.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:18:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:18:24.493+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 13190 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:18:24.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:24.696+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:24.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:24.696+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:24.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:24.696+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:24.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:24.696+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:25.328 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:25.064+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:25.328 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:25.064+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:25.329 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:25.064+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:25.329 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:25.065+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:25.730 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:25.324+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:25.731 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:25.324+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:25.731 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:25.324+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:25.731 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:25.325+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:26.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:25.725+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:26.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:25.725+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:26.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:25.725+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:26.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:25.726+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:26.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:26.086+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:26.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:26.086+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:26.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:26.086+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:26.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:26.087+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:26.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:26.276+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:26.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:26.276+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:26.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:26.276+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:26.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:26.277+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:27.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:26.695+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:27.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:26.695+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:27.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:26.695+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:27.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:26.696+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:27.495 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:27.290+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:27.495 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:27.290+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:27.495 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:27.290+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:27.495 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:27.291+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:27.495 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:27.110+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:27.496 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:27.110+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:27.496 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:27.110+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:27.496 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:27.111+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:27.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:27.697+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:27.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:27.697+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:27.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:27.697+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:27.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:27.698+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:28.519 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:28.244+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:28.519 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:28.244+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:28.519 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:28.244+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:28.519 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:28.244+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:28.519 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:28.072+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:28.520 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:28.072+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:28.520 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:28.072+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:28.520 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:28.072+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:29.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:28.736+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:29.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:28.736+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:29.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:28.736+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:29.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:28.737+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:29.387 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:29.282+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:29.387 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:29.282+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:29.388 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:29.282+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:29.388 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:29.283+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:29.388 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:29.102+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:29.388 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:29.102+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:29.388 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:29.102+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:29.388 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:29.103+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:29.770 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:18:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:18:29.495+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 13284 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:18:30.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:29.766+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:30.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:29.766+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:30.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:29.766+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:30.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:29.766+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:30.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:30.324+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:30.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:30.324+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:30.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:30.324+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:30.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:30.324+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:30.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:30.124+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:30.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:30.124+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:30.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:30.124+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:30.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:30.125+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:31.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:30.718+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:31.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:30.718+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:31.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:30.718+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:31.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:30.719+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:31.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:31.276+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:31.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:31.276+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:31.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:31.276+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:31.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:31.277+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:31.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:31.093+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:31.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:31.093+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:31.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:31.093+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:31.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:31.094+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:32.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:31.740+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:32.033 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:31.740+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:32.033 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:31.740+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:32.033 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:31.741+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:32.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:32.051+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:32.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:32.051+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:32.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:32.051+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:32.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:32.052+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:32.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:32.233+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:32.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:32.233+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:32.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:32.233+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:32.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:32.235+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:33.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:32.774+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:33.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:32.774+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:33.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:32.774+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:33.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:32.775+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:33.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:33.211+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:33.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:33.211+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:33.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:33.211+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:33.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:33.212+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:33.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:33.055+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:33.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:33.055+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:33.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:33.055+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:33.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:33.055+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:34.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:33.744+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:34.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:33.744+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:34.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:33.744+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:34.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:33.745+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:34.501 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:34.053+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:34.501 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:34.053+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:34.501 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:34.053+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:34.501 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:34.053+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:34.501 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:34.211+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:34.502 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:34.211+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:34.502 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:34.211+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:34.502 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:34.212+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:34.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:18:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:18:34.496+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 13378 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:18:34.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:34.729+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:34.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:34.729+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:34.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:34.729+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:34.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:34.729+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:35.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:35.024+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:35.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:35.024+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:35.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:35.024+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:35.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:35.025+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:35.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:35.196+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:35.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:35.196+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:35.282 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:35.196+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:35.282 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:35.197+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:36.017 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:35.717+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:36.018 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:35.717+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:36.018 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:35.717+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:36.018 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:35.718+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:36.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:36.203+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:36.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:36.203+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:36.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:36.203+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:36.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:36.204+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:36.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:36.013+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:36.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:36.013+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:36.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:36.013+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:36.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:36.014+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:37.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:36.761+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:37.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:36.761+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:37.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:36.761+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:37.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:36.762+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:37.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:37.198+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:37.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:37.198+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:37.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:37.198+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:37.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:37.199+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:37.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:37.041+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:37.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:37.041+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:37.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:37.041+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:37.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:37.042+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:38.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:37.781+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:38.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:37.781+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:38.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:37.781+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:38.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:37.783+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:38.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:38.088+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:38.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:38.088+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:38.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:38.088+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:38.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:38.089+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:38.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:38.170+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:38.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:38.170+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:38.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:38.170+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:38.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:38.171+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:39.066 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:38.804+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:39.067 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:38.804+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:39.067 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:38.804+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:39.067 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:38.805+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:39.388 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:39.062+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:39.389 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:39.062+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:39.389 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:39.062+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:39.389 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:39.064+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:39.389 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:39.220+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:39.389 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:39.220+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:39.389 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:39.220+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:39.390 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:39.221+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:39.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:18:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:18:39.498+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 13470 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:18:40.072 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:39.785+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:40.073 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:39.785+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:40.073 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:39.785+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:40.073 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:39.786+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:40.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:40.068+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:40.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:40.068+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:40.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:40.068+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:40.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:40.069+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:40.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:40.178+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:40.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:40.178+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:40.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:40.178+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:40.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:40.178+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:41.096 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:40.829+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:41.096 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:40.829+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:41.097 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:40.829+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:41.097 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:40.830+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:41.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:41.179+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:41.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:41.179+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:41.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:41.179+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:41.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:41.180+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:41.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:41.092+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:41.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:41.092+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:41.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:41.092+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:41.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:41.093+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:42.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:41.851+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:42.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:41.851+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:42.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:41.851+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:42.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:41.852+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:42.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:42.105+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:42.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:42.105+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:42.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:42.105+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:42.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:42.105+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:42.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:42.135+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:42.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:42.135+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:42.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:42.135+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:42.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:42.136+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:43.141 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:42.863+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:43.141 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:42.863+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:43.141 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:42.863+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:43.141 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:42.864+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:43.141 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:43.076+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:43.142 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:43.076+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:43.142 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:43.076+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:43.142 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:43.076+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:43.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:43.137+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:43.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:43.137+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:43.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:43.137+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:43.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:43.138+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:44.170 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:44.053+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:44.170 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:44.053+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:44.170 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:44.053+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:44.170 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:44.054+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:44.171 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:43.910+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:44.171 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:43.910+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:44.171 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:43.910+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:44.171 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:43.911+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:44.503 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:44.165+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:44.503 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:44.165+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:44.503 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:44.165+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:44.503 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:44.166+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:44.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:18:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:18:44.499+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 13565 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:18:45.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:44.944+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:45.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:44.944+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:45.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:44.944+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:45.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:44.944+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:45.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:45.044+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:45.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:45.044+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:45.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:45.044+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:45.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:45.044+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:45.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:45.151+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:45.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:45.151+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:45.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:45.151+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:45.282 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:45.152+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:46.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:46.013+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:46.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:46.013+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:46.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:46.013+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:46.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:46.013+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:46.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:46.181+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:46.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:46.181+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:46.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:46.181+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:46.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:46.182+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:46.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:45.951+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:46.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:45.951+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:46.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:45.951+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:46.282 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:45.952+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:47.207 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:46.998+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:47.207 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:46.998+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:47.207 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:46.998+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:47.207 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:46.998+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:47.207 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:46.937+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:47.208 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:46.937+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:47.208 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:46.937+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:47.208 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:46.938+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:47.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:47.203+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:47.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:47.203+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:47.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:47.203+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:47.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:47.204+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:48.220 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:47.933+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:48.220 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:47.933+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:48.220 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:47.933+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:48.221 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:47.934+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:48.221 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:48.031+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:48.221 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:48.031+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:48.221 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:48.031+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:48.221 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:48.031+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:48.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:48.216+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:48.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:48.216+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:48.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:48.216+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:48.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:48.217+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:49.232 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:48.891+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:49.232 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:48.891+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:49.232 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:48.891+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:49.232 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:48.892+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:49.233 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:49.061+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:49.233 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:49.061+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:49.233 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:49.061+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:49.233 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:49.062+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:49.505 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:49.227+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:49.505 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:49.227+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:49.506 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:49.227+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:49.506 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:49.228+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:49.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:18:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:18:49.501+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 13657 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:18:50.187 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:49.917+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:50.187 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:49.917+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:50.187 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:49.917+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:50.187 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:49.918+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:50.187 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:50.015+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:50.188 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:50.015+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:50.188 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:50.015+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:50.188 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:50.015+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:50.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:50.183+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:50.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:50.183+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:50.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:50.183+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:50.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:50.184+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:51.226 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:50.949+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:51.226 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:50.949+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:51.226 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:50.949+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:51.226 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:50.950+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:51.227 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:51.048+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:51.227 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:51.048+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:51.227 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:51.049+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:51.227 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:51.050+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:51.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:51.222+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:51.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:51.222+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:51.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:51.222+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:51.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:51.222+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:52.023 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:51.916+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:52.024 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:51.916+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:52.024 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:51.916+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:52.024 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:51.917+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:52.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:52.019+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:52.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:52.019+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:52.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:52.019+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:52.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:52.020+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:52.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:52.215+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:52.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:52.215+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:52.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:52.215+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:52.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:52.215+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:53.216 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:52.891+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:53.217 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:52.892+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:53.217 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:52.892+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:53.217 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:52.892+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:53.217 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:52.996+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:53.217 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:52.996+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:53.217 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:52.996+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:53.218 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:52.997+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:53.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:53.212+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:53.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:53.212+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:53.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:53.212+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:53.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:53.213+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:54.223 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:53.940+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:54.223 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:53.940+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:54.223 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:53.940+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:54.223 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:53.941+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:54.223 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:53.953+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:54.223 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:53.953+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:54.224 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:53.953+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:54.224 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:53.954+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:54.506 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:54.218+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:54.507 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:54.218+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:54.507 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:54.218+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:54.507 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:54.220+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:54.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:18:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:18:54.502+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 13751 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:18:55.245 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:54.995+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:55.246 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:54.995+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:55.246 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:54.995+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:55.246 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:54.996+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:55.246 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:54.969+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:55.246 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:54.969+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:55.246 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:54.969+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:55.247 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:54.970+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:55.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:55.241+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:55.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:55.241+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:55.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:55.241+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:55.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:55.242+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:56.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:55.950+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:56.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:55.950+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:56.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:55.950+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:56.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:55.951+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:56.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:56.192+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:56.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:56.192+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:56.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:56.192+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:56.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:56.194+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:56.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:55.972+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:56.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:55.972+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:56.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:55.972+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:56.282 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:55.973+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:57.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:56.954+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:57.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:56.954+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:57.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:56.954+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:57.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:56.955+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:57.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:56.950+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:57.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:56.950+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:57.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:56.950+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:57.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:56.951+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:57.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:57.153+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:57.282 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:57.153+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:57.282 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:57.153+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:57.282 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:57.154+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:58.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:57.946+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:58.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:57.946+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:58.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:57.946+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:58.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:57.946+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:58.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:57.910+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:58.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:57.910+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:58.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:57.910+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:58.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:57.911+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:58.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:58.151+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:58.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:58.151+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:58.282 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:58.151+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:58.282 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:58.152+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:59.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:58.959+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:59.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:58.959+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:59.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:58.959+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:18:59.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:18:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:18:58.960+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:59.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:58.947+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:59.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:58.947+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:59.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:58.948+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:18:59.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:58.949+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:59.388 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:59.121+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:59.388 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:59.122+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:59.388 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:59.122+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:18:59.388 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:18:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:18:59.122+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:18:59.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:18:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:18:59.504+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 13842 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:19:00.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:59.929+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:00.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:59.929+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:00.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:59.929+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:00.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:18:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:18:59.930+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:00.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:00.146+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:00.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:00.146+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:00.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:00.146+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:00.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:00.147+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:00.282 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:00.000+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:00.282 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:00.000+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:00.282 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:00.000+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:00.282 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:00.000+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:01.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:00.962+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:01.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:00.962+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:01.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:00.962+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:01.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:00.963+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:01.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:01.102+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:01.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:01.102+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:01.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:01.102+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:01.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:01.102+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:01.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:01.006+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:01.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:01.006+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:01.282 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:01.006+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:01.282 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:01.007+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:02.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:01.960+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:02.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:01.960+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:02.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:01.960+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:02.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:01.960+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:02.282 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:02.076+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:02.282 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:02.076+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:02.282 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:02.076+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:02.283 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:02.077+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:02.283 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:02.055+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:02.283 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:02.055+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:02.284 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:02.055+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:02.284 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:02.056+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:03.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:03.009+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:03.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:03.009+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:03.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:03.009+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:03.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:03.010+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:03.282 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:03.118+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:03.282 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:03.118+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:03.282 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:03.118+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:03.283 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:03.118+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:03.283 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:02.933+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:03.283 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:02.933+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:03.284 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:02.933+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:03.284 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:02.934+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:04.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:04.168+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:04.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:04.168+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:04.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:04.168+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:04.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:04.169+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:04.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:03.960+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:04.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:03.960+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:04.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:03.960+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:04.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:03.961+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:04.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:04.053+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:04.282 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:04.053+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:04.282 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:04.053+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:04.282 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:04.054+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:04.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:19:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:19:04.505+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 13937 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:19:05.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:05.137+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:05.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:05.137+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:05.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:05.137+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:05.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:05.138+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:05.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:05.076+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:05.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:05.076+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:05.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:05.076+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:05.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:05.077+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:05.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:04.930+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:05.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:04.930+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:05.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:04.930+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:05.282 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:04.931+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:06.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:06.152+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:06.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:06.152+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:06.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:06.152+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:06.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:06.153+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:06.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:05.929+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:06.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:05.929+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:06.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:05.929+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:06.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:05.930+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:06.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:06.122+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:06.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:06.122+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:06.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:06.122+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:06.282 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:06.123+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:07.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:07.196+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:07.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:07.196+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:07.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:07.196+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:07.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:07.197+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:07.282 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:07.172+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:07.282 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:07.172+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:07.282 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:07.172+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:07.282 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:07.173+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:07.283 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:06.969+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:07.283 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:06.969+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:07.283 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:06.969+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:07.283 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:06.970+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:08.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:08.165+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:08.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:08.165+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:08.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:08.165+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:08.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:08.166+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:08.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:08.213+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:08.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:08.213+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:08.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:08.213+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:08.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:08.214+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:08.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:07.971+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:08.282 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:07.971+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:08.282 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:07.971+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:08.282 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:07.972+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:09.211 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:09.147+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:09.211 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:09.147+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:09.211 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:09.147+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:09.211 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:09.148+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:09.212 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:08.945+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:09.212 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:08.945+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:09.212 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:08.945+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:09.212 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:08.946+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:09.511 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:09.206+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:09.512 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:09.206+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:09.512 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:09.206+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:09.512 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:09.207+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:09.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:19:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:19:09.507+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 14029 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:19:10.243 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:09.934+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:10.244 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:09.934+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:10.244 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:09.934+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:10.244 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:09.934+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:10.245 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:10.129+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:10.245 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:10.129+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:10.245 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:10.129+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:10.246 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:10.129+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:10.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:10.239+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:10.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:10.239+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:10.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:10.239+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:10.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:10.240+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:11.241 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:11.146+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:11.242 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:11.146+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:11.242 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:11.146+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:11.242 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:11.147+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:11.242 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:10.904+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:11.242 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:10.904+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:11.243 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:10.904+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:11.243 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:10.904+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:11.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:11.237+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:11.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:11.237+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:11.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:11.237+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:11.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:11.238+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:12.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:11.915+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:12.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:11.915+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:12.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:11.915+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:12.031 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:11.915+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:12.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:12.106+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:12.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:12.106+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:12.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:12.106+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:12.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:12.107+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:12.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:12.191+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:12.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:12.191+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:12.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:12.191+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:12.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:12.192+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:13.242 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:12.944+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:13.242 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:12.944+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:13.242 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:12.944+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:13.243 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:12.945+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:13.243 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:13.116+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:13.243 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:13.116+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:13.243 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:13.116+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:13.243 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:13.116+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:13.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:13.238+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:13.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:13.238+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:13.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:13.238+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:13.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:13.239+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:14.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:13.972+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:14.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:13.972+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:14.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:13.972+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:14.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:13.973+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:14.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:14.137+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:14.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:14.137+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:14.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:14.137+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:14.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:14.138+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:14.598 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:19:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:19:14.509+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 14122 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:19:14.599 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:14.282+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:14.599 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:14.282+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:14.599 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:14.282+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:14.599 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:14.283+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:15.271 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:15.159+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:15.272 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:15.159+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:15.272 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:15.159+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:15.272 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:15.160+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:15.272 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:14.972+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:15.272 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:14.972+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:15.272 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:14.972+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:15.272 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:14.973+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:15.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:15.267+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:15.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:15.267+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:15.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:15.267+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:15.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:15.268+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:16.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:16.163+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:16.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:16.163+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:16.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:16.163+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:16.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:16.164+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:16.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:16.240+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:16.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:16.240+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:16.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:16.240+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:16.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:16.241+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:16.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:16.003+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:16.282 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:16.003+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:16.282 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:16.003+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:16.282 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:16.004+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:17.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:17.142+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:17.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:17.142+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:17.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:17.142+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:17.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:17.142+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:17.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:17.214+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:17.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:17.214+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:17.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:17.214+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:17.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:17.215+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:17.282 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:16.969+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:17.282 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:16.969+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:17.282 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:16.969+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:17.282 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:16.970+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:18.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:18.192+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:18.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:18.192+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:18.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:18.192+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:18.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:18.193+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:18.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:18.109+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:18.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:18.109+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:18.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:18.109+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:18.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:18.109+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:18.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:17.959+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:18.282 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:17.959+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:18.282 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:17.959+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:18.282 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:17.960+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:19.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:18.970+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:19.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:18.970+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:19.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:18.970+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:19.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:18.971+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:19.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:19.121+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:19.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:19.121+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:19.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:19.121+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:19.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:19.122+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:19.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:19.160+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:19.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:19.160+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:19.282 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:19.160+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:19.282 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:19.161+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:19.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:19:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:19:19.510+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 14213 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:19:20.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:20.154+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:20.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:20.154+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:20.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:20.154+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:20.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:20.155+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:20.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:20.137+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:20.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:20.137+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:20.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:20.137+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:20.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:20.138+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:20.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:19.940+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:20.282 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:19.940+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:20.282 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:19.940+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:20.282 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:19.940+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:21.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:20.983+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:21.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:20.983+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:21.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:20.983+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:21.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:20.984+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:21.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:21.179+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:21.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:21.179+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:21.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:21.179+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:21.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:21.180+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:21.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:21.106+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:21.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:21.106+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:21.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:21.106+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:21.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:21.107+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:22.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:22.189+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:22.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:22.189+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:22.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:22.189+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:22.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:22.189+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:22.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:22.111+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:22.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:22.111+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:22.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:22.111+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:22.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:22.112+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:22.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:22.023+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:22.282 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:22.023+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:22.282 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:22.023+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:22.282 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:22.023+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:23.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:23.178+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:23.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:23.178+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:23.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:23.178+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:23.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:23.179+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:23.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:23.084+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:23.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:23.084+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:23.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:23.084+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:23.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:23.085+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:23.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:23.031+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:23.532 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:23.031+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:23.532 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:23.031+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:23.532 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:23.032+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:24.516 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:24.075+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:24.516 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:24.075+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:24.516 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:24.075+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:24.516 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:24.076+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:24.517 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:24.048+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:24.517 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:24.048+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:24.517 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:24.048+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:24.517 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:24.049+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:24.517 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:24.181+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:24.518 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:24.181+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:24.518 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:24.181+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:24.518 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:24.181+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:24.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:19:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:19:24.512+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 14298 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:19:25.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:25.144+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:25.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:25.144+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:25.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:25.144+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:25.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:25.145+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:25.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:25.112+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:25.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:25.112+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:25.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:25.112+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:25.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:25.113+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:25.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:25.080+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:25.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:25.080+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:25.532 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:25.080+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:25.532 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:25.081+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:26.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:26.073+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:26.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:26.073+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:26.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:26.073+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:26.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:26.074+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:26.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:26.166+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:26.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:26.166+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:26.532 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:26.166+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:26.532 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:26.167+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:26.532 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:26.091+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:26.532 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:26.091+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:26.532 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:26.091+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:26.532 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:26.092+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:27.137 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:27.038+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:27.137 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:27.038+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:27.137 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:27.038+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:27.137 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:27.039+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:27.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:27.174+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:27.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:27.174+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:27.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:27.174+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:27.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:27.175+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:27.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:27.133+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:27.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:27.133+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:27.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:27.133+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:27.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:27.134+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:28.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:28.160+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:28.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:28.160+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:28.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:28.160+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:28.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:28.161+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:28.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:28.179+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:28.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:28.179+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:28.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:28.179+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:28.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:28.179+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:28.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:28.020+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:28.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:28.020+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:28.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:28.020+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:28.282 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:28.021+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:29.388 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:29.048+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:29.388 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:29.048+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:29.388 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:29.048+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:29.388 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:29.049+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:29.389 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:29.213+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:29.389 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:29.213+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:29.389 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:29.213+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:29.389 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:29.214+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:29.389 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:29.164+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:29.390 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:29.164+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:29.390 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:29.164+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:29.390 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:29.165+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:29.641 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:19:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:19:29.513+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 14377 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:19:30.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:30.143+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:30.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:30.143+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:30.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:30.143+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:30.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:30.143+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:30.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:30.096+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:30.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:30.096+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:30.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:30.096+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:30.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:30.097+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:30.532 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:30.171+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:30.532 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:30.171+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:30.532 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:30.171+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:30.532 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:30.172+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:31.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:31.150+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:31.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:31.150+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:31.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:31.150+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:31.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:31.151+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:31.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:31.090+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:31.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:31.090+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:31.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:31.090+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:31.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:31.091+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:31.532 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:31.126+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:31.532 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:31.126+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:31.532 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:31.126+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:31.532 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:31.127+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:32.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:32.079+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:32.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:32.079+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:32.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:32.079+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:32.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:32.080+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:32.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:32.150+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:32.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:32.150+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:32.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:32.150+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:32.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:32.151+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:32.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:32.164+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:32.532 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:32.164+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:32.532 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:32.164+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:32.532 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:32.165+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:33.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:33.157+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:33.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:33.157+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:33.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:33.157+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:33.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:33.158+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:33.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:33.062+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:33.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:33.062+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:33.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:33.062+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:33.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:33.062+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:33.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:33.123+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:33.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:33.123+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:33.532 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:33.123+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:33.532 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:33.124+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:34.519 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:34.170+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:34.519 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:34.170+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:34.519 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:34.170+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:34.519 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:34.170+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:34.520 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:34.135+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:34.520 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:34.135+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:34.520 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:34.135+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:34.520 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:34.136+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:34.520 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:34.101+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:34.521 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:34.101+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:34.521 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:34.101+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:34.521 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:34.101+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:34.780 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:19:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:19:34.515+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 14459 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:19:35.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:35.086+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:35.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:35.086+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:35.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:35.086+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:35.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:35.086+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:35.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:35.148+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:35.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:35.148+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:35.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:35.148+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:35.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:35.148+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:35.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:35.125+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:35.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:35.125+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:35.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:35.125+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:35.532 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:35.125+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:36.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:36.040+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:36.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:36.040+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:36.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:36.040+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:36.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:36.040+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:36.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:36.085+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:36.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:36.085+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:36.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:36.085+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:36.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:36.086+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:36.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:36.145+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:36.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:36.145+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:36.532 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:36.145+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:36.532 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:36.146+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:37.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:37.173+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:37.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:37.173+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:37.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:37.173+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:37.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:37.174+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:37.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:37.088+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:37.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:37.088+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:37.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:37.088+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:37.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:37.089+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:37.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:37.039+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:37.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:37.039+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:37.532 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:37.039+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:37.532 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:37.040+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:38.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:38.224+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:38.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:38.224+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:38.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:38.224+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:38.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:38.224+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:38.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:38.012+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:38.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:38.012+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:38.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:38.012+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:38.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:38.012+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:38.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:38.106+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:38.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:38.106+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:38.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:38.106+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:38.282 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:38.107+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:39.235 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:39.066+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:39.235 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:39.066+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:39.236 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:39.066+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:39.236 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:39.067+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:39.236 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:39.206+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:39.236 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:39.206+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:39.236 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:39.206+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:39.236 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:39.207+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:39.237 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:38.963+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:39.237 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:38.963+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:39.237 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:38.963+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:39.237 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:38.964+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:39.238 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-11T23:19:39.238 WARNING:tasks.kclient:Ordinary umount failed, forcing... 2024-03-11T23:19:39.239 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-11T23:19:39.239 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-11T23:19:39.256 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-11T23:19:39.256 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-11T23:19:39.312 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-11T23:19:39.779 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:19:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:19:39.516+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 14539 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:19:40.228 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:39.972+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:40.229 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:39.972+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:40.229 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:39.972+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:40.229 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:39.972+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:40.229 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:40.116+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:40.229 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:40.116+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:40.229 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:40.116+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:40.230 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:40.116+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:40.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:40.224+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:40.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:40.224+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:40.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:40.224+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:40.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:40.226+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:41.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:41.227+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:41.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:41.227+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:41.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:41.227+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:41.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:41.228+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:41.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:41.021+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:41.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:41.021+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:41.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:41.021+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:41.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:41.022+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:41.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:41.118+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:41.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:41.118+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:41.282 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:41.118+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:41.282 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:41.118+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:42.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:42.012+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:42.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:42.012+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:42.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:42.012+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:42.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:42.013+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:42.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:42.092+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:42.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:42.092+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:42.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:42.092+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:42.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:42.092+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:42.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:42.231+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:42.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:42.231+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:42.282 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:42.231+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:42.282 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:42.232+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:43.237 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:43.131+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:43.238 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:43.131+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:43.238 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:43.131+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:43.238 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:43.131+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:43.239 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:42.976+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:43.239 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:42.976+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:43.239 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:42.976+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:43.239 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:42.976+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:43.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:43.233+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:43.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:43.233+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:43.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:43.233+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:43.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:43.234+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:44.203 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:43.941+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:44.203 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:43.941+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:44.203 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:43.941+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:44.203 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:43.942+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:44.203 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:44.132+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:44.204 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:44.132+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:44.204 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:44.132+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:44.204 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:44.133+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:44.522 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:44.199+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:44.522 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:44.199+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:44.522 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:44.199+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:44.522 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:44.200+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:44.779 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:19:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:19:44.518+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 14621 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:19:45.160 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:44.931+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:45.161 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:44.931+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:45.161 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:44.931+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:45.161 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:44.932+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:45.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:45.217+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:45.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:45.217+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:45.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:45.217+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:45.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:45.218+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:45.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:45.156+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:45.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:45.157+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:45.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:45.157+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:45.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:45.158+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:46.203 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:45.895+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:46.203 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:45.895+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:46.203 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:45.895+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:46.203 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:45.896+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:46.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:46.199+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:46.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:46.199+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:46.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:46.199+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:46.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:46.199+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:46.533 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:46.258+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:46.533 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:46.258+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:46.533 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:46.258+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:46.534 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:46.258+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:47.207 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:46.912+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:47.207 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:46.912+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:47.207 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:46.912+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:47.208 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:46.913+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:47.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:47.203+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:47.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:47.203+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:47.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:47.203+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:47.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:47.204+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:47.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:47.237+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:47.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:47.237+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:47.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:47.237+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:47.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:47.238+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:48.229 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:47.922+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:48.229 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:47.922+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:48.229 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:47.922+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:48.229 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:47.923+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:48.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:48.225+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:48.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:48.225+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:48.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:48.225+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:48.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:48.226+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:48.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:48.240+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:48.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:48.240+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:48.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:48.240+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:48.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:48.240+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:49.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:49.197+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:49.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:49.197+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:49.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:49.197+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:49.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:49.197+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:49.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:48.951+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:49.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:48.951+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:49.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:48.951+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:49.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:48.952+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:49.548 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:19:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:19:49.519+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 14701 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:19:49.548 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:49.284+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:49.548 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:49.284+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:49.548 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:49.284+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:49.549 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:49.284+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:50.161 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:49.904+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:50.161 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:49.904+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:50.162 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:49.904+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:50.162 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:49.905+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:50.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:50.157+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:50.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:50.157+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:50.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:50.157+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:50.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:50.158+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:50.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:50.319+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:50.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:50.319+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:50.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:50.319+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:50.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:50.320+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:51.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:50.918+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:51.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:50.918+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:51.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:50.918+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:51.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:50.918+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:51.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:51.151+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:51.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:51.151+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:51.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:51.151+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:51.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:51.152+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:51.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:51.273+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:51.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:51.273+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:51.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:51.273+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:51.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:51.274+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:52.228 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:51.963+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:52.229 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:51.963+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:52.229 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:51.963+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:52.229 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:51.964+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:52.229 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:52.135+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:52.229 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:52.135+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:52.230 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:52.135+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:52.230 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:52.136+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:52.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:52.224+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:52.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:52.224+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:52.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:52.224+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:52.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:52.225+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:53.245 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:53.139+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:53.246 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:53.139+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:53.246 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:53.139+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:53.246 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:53.140+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:53.246 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:52.943+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:53.246 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:52.944+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:53.247 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:52.944+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:53.247 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:52.944+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:53.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:53.242+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:53.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:53.242+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:53.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:53.242+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:53.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:53.243+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:54.265 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:53.976+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:54.265 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:53.976+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:54.265 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:53.976+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:54.265 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:53.977+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:54.265 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:54.162+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:54.265 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:54.162+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:54.266 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:54.162+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:54.266 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:54.163+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:54.525 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:54.261+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:54.525 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:54.261+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:54.525 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:54.261+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:54.525 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:54.261+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:54.779 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:19:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:19:54.521+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 14783 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:19:55.267 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:55.003+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:55.268 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:55.003+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:55.268 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:55.003+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:55.268 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:55.004+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:55.268 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:55.190+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:55.268 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:55.190+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:55.269 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:55.190+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:55.269 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:55.191+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:55.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:55.264+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:55.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:55.264+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:55.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:55.264+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:55.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:55.264+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:56.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:56.213+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:56.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:56.213+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:56.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:56.213+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:56.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:56.214+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:56.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:55.989+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:56.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:55.989+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:56.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:55.989+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:56.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:55.990+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:56.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:56.230+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:56.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:56.230+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:56.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:56.230+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:56.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:56.231+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:57.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:57.185+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:57.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:57.185+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:57.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:57.185+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:57.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:57.186+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:57.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:56.969+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:57.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:56.969+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:57.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:56.969+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:57.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:56.970+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:57.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:57.197+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:57.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:57.197+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:57.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:57.197+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:57.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:57.198+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:58.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:57.976+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:58.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:57.976+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:58.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:57.976+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:58.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:57.976+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:58.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:58.172+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:58.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:58.172+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:58.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:58.172+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:58.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:58.173+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:58.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:58.159+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:58.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:58.159+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:58.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:58.159+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:58.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:58.160+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:59.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:59.205+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:59.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:59.205+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:59.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:59.205+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:19:59.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:19:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:19:59.206+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:59.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:59.007+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:59.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:59.007+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:59.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:59.007+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:19:59.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:59.008+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:59.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:59.220+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:59.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:59.220+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:59.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:59.221+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:19:59.282 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:19:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:19:59.221+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:19:59.650 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:19:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:19:59.522+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 14863 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:20:00.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:00.187+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:00.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:00.187+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:00.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:00.187+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:00.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:00.188+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:00.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:59.976+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:00.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:59.976+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:00.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:59.976+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:00.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:19:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:19:59.977+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:00.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:00.173+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:00.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:00.173+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:00.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:00.173+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:00.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:00.174+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:01.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:01.143+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:01.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:01.143+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:01.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:01.143+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:01.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:01.144+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:01.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:01.141+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:01.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:01.141+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:01.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:01.141+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:01.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:01.142+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:01.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:01.000+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:01.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:01.000+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:01.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:01.000+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:01.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:01.001+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:02.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:01.994+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:02.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:01.994+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:02.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:01.994+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:02.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:01.994+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:02.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:02.103+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:02.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:02.103+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:02.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:02.103+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:02.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:02.104+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:02.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:02.121+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:02.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:02.121+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:02.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:02.121+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:02.282 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:02.122+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:03.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:03.168+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:03.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:03.168+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:03.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:03.168+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:03.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:03.169+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:03.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:02.987+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:03.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:02.987+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:03.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:02.987+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:03.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:02.988+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:03.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:03.089+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:03.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:03.089+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:03.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:03.089+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:03.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:03.090+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:04.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:04.107+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:04.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:04.107+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:04.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:04.107+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:04.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:04.108+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:04.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:04.178+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:04.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:04.178+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:04.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:04.178+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:04.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:04.179+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:04.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:03.982+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:04.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:03.982+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:04.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:03.982+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:04.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:03.983+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:04.779 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:20:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:20:04.524+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 14945 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:20:05.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:05.206+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:05.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:05.206+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:05.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:05.206+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:05.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:05.207+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:05.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:05.018+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:05.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:05.018+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:05.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:05.018+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:05.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:05.018+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:05.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:05.131+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:05.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:05.131+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:05.282 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:05.131+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:05.282 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:05.132+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:06.205 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:05.989+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:06.205 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:05.989+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:06.205 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:05.989+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:06.205 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:05.990+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:06.206 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:06.146+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:06.206 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:06.146+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:06.206 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:06.146+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:06.206 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:06.147+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:06.206 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:06.199+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:06.207 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:06.199+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:06.207 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:06.199+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:06.207 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:06.200+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:07.102 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:06.966+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:07.102 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:06.966+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:07.102 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:06.966+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:07.102 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:06.966+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:07.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:07.151+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:07.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:07.151+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:07.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:07.151+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:07.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:07.151+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:07.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:07.098+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:07.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:07.098+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:07.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:07.098+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:07.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:07.099+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:08.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:08.013+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:08.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:08.013+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:08.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:08.013+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:08.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:08.013+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:08.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:08.101+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:08.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:08.101+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:08.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:08.101+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:08.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:08.102+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:08.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:08.183+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:08.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:08.183+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:08.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:08.183+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:08.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:08.183+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:09.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:08.996+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:09.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:08.996+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:09.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:08.996+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:09.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:08.997+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:09.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:09.084+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:09.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:09.084+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:09.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:09.084+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:09.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:09.085+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:09.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:09.193+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:09.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:09.193+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:09.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:09.193+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:09.282 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:09.195+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:09.653 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:20:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:20:09.525+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 15025 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:20:10.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:10.024+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:10.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:10.024+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:10.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:10.024+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:10.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:10.025+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:10.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:10.122+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:10.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:10.122+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:10.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:10.122+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:10.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:10.122+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:10.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:10.187+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:10.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:10.187+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:10.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:10.187+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:10.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:10.187+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:11.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:11.014+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:11.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:11.014+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:11.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:11.014+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:11.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:11.016+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:11.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:11.154+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:11.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:11.154+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:11.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:11.154+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:11.281 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:11.154+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:11.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:11.126+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:11.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:11.126+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:11.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:11.126+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:11.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:11.127+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:12.529 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:12.057+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:12.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:12.057+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:12.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:12.057+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:12.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:12.058+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:12.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:12.167+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:12.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:12.167+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:12.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:12.167+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:12.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:12.168+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:12.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:12.149+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:12.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:12.149+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:12.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:12.149+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:12.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:12.150+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:13.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:13.144+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:13.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:13.144+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:13.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:13.144+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:13.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:13.144+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:13.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:13.170+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:13.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:13.170+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:13.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:13.170+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:13.281 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:13.171+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:13.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:13.076+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:13.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:13.076+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:13.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:13.076+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:13.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:13.077+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:14.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:14.177+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:14.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:14.177+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:14.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:14.177+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:14.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:14.178+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:14.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:14.070+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:14.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:14.070+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:14.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:14.070+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:14.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:14.071+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:14.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:14.188+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:14.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:14.188+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:14.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:14.188+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:14.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:14.189+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:15.029 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:20:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:20:14.527+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 15106 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:20:15.529 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:15.031+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:15.529 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:15.031+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:15.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:15.031+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:15.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:15.032+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:15.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:15.185+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:15.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:15.185+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:15.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:15.185+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:15.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:15.186+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:15.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:15.195+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:15.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:15.195+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:15.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:15.195+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:15.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:15.196+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:16.529 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:16.041+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:16.529 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:16.041+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:16.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:16.041+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:16.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:16.042+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:16.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:16.171+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:16.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:16.171+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:16.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:16.171+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:16.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:16.172+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:16.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:16.245+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:16.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:16.245+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:16.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:16.245+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:16.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:16.246+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:17.266 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:17.214+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:17.267 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:17.214+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:17.267 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:17.214+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:17.267 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:17.215+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:17.267 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:17.005+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:17.267 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:17.005+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:17.267 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:17.005+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:17.268 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:17.006+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:17.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:17.263+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:17.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:17.263+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:17.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:17.263+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:17.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:17.263+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:18.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:18.014+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:18.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:18.014+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:18.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:18.014+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:18.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:18.015+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:18.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:18.204+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:18.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:18.204+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:18.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:18.204+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:18.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:18.205+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:18.779 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:18.307+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:18.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:18.307+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:18.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:18.307+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:18.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:18.308+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:19.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:19.212+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:19.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:19.212+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:19.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:19.212+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:19.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:19.213+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:19.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:18.990+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:19.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:18.990+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:19.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:18.990+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:19.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:18.991+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:19.657 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:20:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:20:19.528+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 15185 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:20:19.657 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:19.307+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:19.657 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:19.307+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:19.658 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:19.307+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:19.658 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:19.308+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:20.246 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:19.969+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:20.247 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:19.969+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:20.247 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:19.969+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:20.247 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:19.970+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:20.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:20.243+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:20.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:20.243+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:20.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:20.243+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:20.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:20.244+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:20.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:20.342+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:20.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:20.342+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:20.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:20.342+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:20.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:20.343+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:21.233 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:20.948+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:21.233 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:20.948+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:21.233 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:20.948+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:21.234 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:20.949+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:21.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:21.229+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:21.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:21.229+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:21.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:21.229+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:21.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:21.231+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:21.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:21.314+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:21.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:21.314+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:21.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:21.314+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:21.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:21.315+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:22.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:21.926+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:22.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:21.926+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:22.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:21.926+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:22.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:21.926+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:22.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:22.207+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:22.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:22.207+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:22.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:22.207+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:22.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:22.208+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:22.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:22.341+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:22.534 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:22.341+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:22.534 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:22.341+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:22.534 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:22.342+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:23.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:22.955+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:23.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:22.955+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:23.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:22.955+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:23.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:22.956+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:23.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:23.198+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:23.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:23.198+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:23.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:23.198+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:23.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:23.199+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:23.779 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:23.346+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:23.779 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:23.346+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:23.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:23.346+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:23.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:23.346+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:24.223 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:23.960+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:24.223 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:23.960+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:24.224 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:23.960+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:24.224 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:23.962+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:24.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:24.220+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:24.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:24.220+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:24.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:24.220+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:24.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:24.220+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:24.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:24.349+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:24.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:24.349+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:24.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:24.349+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:24.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:24.350+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:24.928 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:20:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:20:24.530+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 15267 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:20:25.231 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:24.925+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:25.231 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:24.925+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:25.231 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:24.925+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:25.231 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:24.926+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:25.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:25.227+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:25.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:25.227+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:25.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:25.227+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:25.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:25.228+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:25.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:25.360+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:25.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:25.360+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:25.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:25.360+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:25.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:25.361+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:26.274 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:25.901+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:26.274 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:25.901+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:26.274 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:25.901+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:26.274 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:25.902+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:26.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:26.270+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:26.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:26.270+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:26.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:26.270+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:26.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:26.271+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:26.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:26.358+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:26.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:26.358+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:26.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:26.358+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:26.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:26.359+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:27.226 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:26.888+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:27.226 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:26.888+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:27.227 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:26.888+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:27.227 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:26.889+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:27.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:27.222+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:27.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:27.222+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:27.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:27.222+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:27.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:27.223+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:27.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:27.312+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:27.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:27.312+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:27.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:27.312+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:27.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:27.313+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:28.181 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:27.854+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:28.181 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:27.854+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:28.181 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:27.854+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:28.182 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:27.855+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:28.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:28.177+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:28.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:28.177+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:28.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:28.177+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:28.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:28.178+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:28.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:28.314+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:28.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:28.314+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:28.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:28.314+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:28.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:28.315+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:29.166 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:28.835+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:29.167 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:28.835+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:29.167 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:28.835+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:29.167 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:28.836+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:29.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:29.351+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:29.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:29.351+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:29.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:29.351+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:29.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:29.352+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:29.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:29.162+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:29.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:29.162+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:29.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:29.162+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:29.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:29.163+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:29.855 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:20:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:20:29.531+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 15345 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:20:30.164 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:29.852+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:30.164 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:29.852+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:30.164 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:29.852+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:30.164 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:29.852+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:30.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:30.160+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:30.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:30.160+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:30.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:30.160+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:30.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:30.161+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:30.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:30.310+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:30.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:30.310+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:30.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:30.310+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:30.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:30.311+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:31.165 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:30.853+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:31.165 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:30.853+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:31.165 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:30.853+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:31.166 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:30.854+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:31.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:31.357+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:31.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:31.357+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:31.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:31.357+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:31.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:31.357+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:31.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:31.161+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:31.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:31.161+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:31.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:31.161+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:31.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:31.162+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:32.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:31.874+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:32.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:31.874+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:32.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:31.874+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:32.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:31.874+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:32.409 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:32.209+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:32.409 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:32.209+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:32.409 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:32.209+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:32.409 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:32.210+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:32.779 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:32.405+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:32.779 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:32.405+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:32.779 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:32.405+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:32.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:32.406+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:33.245 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:32.901+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:33.245 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:32.901+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:33.245 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:32.901+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:33.245 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:32.902+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:33.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:33.383+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:33.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:33.383+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:33.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:33.383+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:33.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:33.384+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:33.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:33.241+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:33.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:33.241+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:33.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:33.241+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:33.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:33.242+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:34.218 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:33.856+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:34.218 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:33.857+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:34.218 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:33.857+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:34.218 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:33.857+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:34.469 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:34.430+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:34.470 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:34.430+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:34.470 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:34.430+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:34.470 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:34.430+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:34.470 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:34.214+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:34.471 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:34.214+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:34.471 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:34.214+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:34.471 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:34.215+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:34.779 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:20:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:20:34.533+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 15425 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:20:35.250 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:34.884+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:35.250 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:34.884+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:35.250 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:34.884+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:35.251 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:34.885+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:35.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:35.389+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:35.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:35.389+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:35.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:35.389+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:35.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:35.390+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:35.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:35.246+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:35.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:35.246+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:35.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:35.246+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:35.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:35.247+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:36.270 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:35.843+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:36.270 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:35.843+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:36.270 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:35.843+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:36.270 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:35.844+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:36.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:36.266+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:36.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:36.266+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:36.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:36.266+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:36.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:36.267+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:36.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:36.342+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:36.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:36.342+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:36.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:36.342+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:36.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:36.343+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:37.259 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:36.869+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:37.259 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:36.869+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:37.260 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:36.869+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:37.260 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:36.869+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:37.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:37.294+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:37.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:37.294+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:37.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:37.294+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:37.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:37.295+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:37.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:37.255+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:37.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:37.255+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:37.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:37.255+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:37.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:37.256+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:38.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:37.911+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:38.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:37.911+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:38.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:37.911+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:38.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:37.912+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:38.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:38.303+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:38.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:38.303+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:38.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:38.303+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:38.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:38.304+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:38.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:38.273+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:38.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:38.273+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:38.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:38.273+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:38.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:38.274+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:39.267 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:38.868+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:39.268 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:38.868+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:39.268 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:38.868+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:39.268 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:38.869+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:39.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:39.264+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:39.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:39.264+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:39.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:39.264+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:39.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:39.265+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:39.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:39.311+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:39.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:39.311+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:39.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:39.311+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:39.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:39.311+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:39.827 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:20:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:20:39.535+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 15503 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:20:40.234 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:39.824+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:40.235 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:39.824+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:40.235 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:39.824+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:40.235 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:39.825+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:40.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:40.231+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:40.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:40.231+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:40.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:40.231+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:40.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:40.232+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:40.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:40.342+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:40.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:40.342+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:40.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:40.342+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:40.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:40.343+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:41.227 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:40.855+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:41.227 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:40.855+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:41.227 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:40.855+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:41.228 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:40.856+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:41.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:41.224+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:41.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:41.224+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:41.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:41.224+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:41.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:41.224+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:41.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:41.382+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:41.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:41.382+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:41.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:41.382+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:41.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:41.383+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:42.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:41.868+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:42.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:41.868+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:42.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:41.868+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:42.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:41.869+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:42.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:42.375+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:42.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:42.375+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:42.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:42.375+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:42.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:42.376+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:42.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:42.203+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:42.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:42.203+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:42.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:42.203+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:42.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:42.204+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:43.252 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:42.910+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:43.252 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:42.910+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:43.252 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:42.910+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:43.253 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:42.911+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:43.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:43.249+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:43.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:43.249+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:43.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:43.249+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:43.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:43.249+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:43.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:43.383+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:43.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:43.383+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:43.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:43.383+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:43.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:43.383+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:44.203 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:43.882+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:44.203 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:43.882+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:44.203 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:43.882+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:44.203 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:43.883+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:44.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:44.200+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:44.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:44.200+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:44.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:44.200+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:44.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:44.201+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:44.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:44.391+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:44.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:44.391+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:44.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:44.391+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:44.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:44.392+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:44.841 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:20:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:20:44.537+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 15584 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:20:45.156 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:44.837+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:45.157 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:44.837+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:45.157 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:44.837+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:45.157 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:44.838+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:45.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:45.153+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:45.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:45.153+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:45.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:45.153+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:45.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:45.154+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:45.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:45.375+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:45.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:45.375+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:45.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:45.375+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:45.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:45.376+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:46.160 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:45.791+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:46.160 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:45.791+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:46.161 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:45.791+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:46.161 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:45.791+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:46.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:46.157+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:46.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:46.157+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:46.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:46.157+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:46.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:46.158+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:46.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:46.329+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:46.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:46.329+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:46.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:46.329+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:46.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:46.330+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:47.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:46.751+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:47.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:46.751+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:47.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:46.751+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:47.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:46.752+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:47.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:47.192+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:47.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:47.192+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:47.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:47.192+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:47.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:47.193+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:47.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:47.379+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:47.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:47.379+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:47.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:47.379+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:47.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:47.380+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:48.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:47.719+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:48.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:47.719+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:48.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:47.719+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:48.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:47.720+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:48.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:48.182+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:48.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:48.182+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:48.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:48.182+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:48.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:48.182+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:48.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:48.347+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:48.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:48.347+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:48.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:48.347+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:48.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:48.348+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:49.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:48.765+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:49.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:48.766+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:49.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:48.766+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:49.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:48.766+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:49.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:49.211+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:49.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:49.211+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:49.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:49.211+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:49.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:49.212+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:49.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:49.382+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:49.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:49.382+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:49.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:49.382+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:49.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:49.383+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:49.830 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:20:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:20:49.538+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 15663 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:20:49.830 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:49.734+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:49.831 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:49.734+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:49.831 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:49.734+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:49.831 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:49.735+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:50.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:50.179+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:50.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:50.179+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:50.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:50.179+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:50.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:50.180+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:50.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:50.338+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:50.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:50.338+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:50.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:50.338+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:50.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:50.339+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:51.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:50.703+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:51.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:50.703+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:51.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:50.703+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:51.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:50.703+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:51.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:51.189+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:51.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:51.189+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:51.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:51.189+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:51.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:51.190+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:51.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:51.291+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:51.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:51.291+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:51.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:51.291+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:51.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:51.292+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:52.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:51.748+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:52.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:51.748+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:52.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:51.748+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:52.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:51.749+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:52.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:52.220+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:52.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:52.220+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:52.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:52.220+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:52.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:52.220+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:52.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:52.263+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:52.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:52.263+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:52.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:52.263+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:52.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:52.264+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:53.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:52.748+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:53.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:52.748+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:53.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:52.748+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:53.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:52.749+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:53.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:53.217+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:53.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:53.217+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:53.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:53.217+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:53.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:53.218+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:53.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:53.284+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:53.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:53.284+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:53.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:53.284+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:53.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:53.285+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:54.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:53.737+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:54.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:53.737+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:54.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:53.737+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:54.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:53.738+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:54.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:54.289+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:54.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:54.289+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:54.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:54.289+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:54.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:54.289+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:54.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:54.230+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:54.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:54.230+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:54.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:54.230+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:54.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:54.231+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:55.029 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:20:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:20:54.540+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 15744 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:20:55.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:54.772+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:55.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:54.772+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:55.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:54.772+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:55.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:54.773+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:55.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:55.198+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:55.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:55.198+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:55.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:55.198+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:55.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:55.199+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:55.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:55.247+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:55.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:55.247+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:55.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:55.247+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:55.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:55.247+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:56.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:55.734+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:56.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:55.734+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:56.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:55.734+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:56.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:55.735+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:56.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:56.163+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:56.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:56.163+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:56.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:56.163+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:56.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:56.164+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:56.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:56.207+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:56.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:56.207+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:56.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:56.207+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:56.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:56.207+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:57.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:56.714+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:57.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:56.714+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:57.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:56.714+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:57.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:56.715+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:57.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:57.121+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:57.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:57.121+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:57.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:57.121+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:57.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:57.122+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:57.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:57.208+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:57.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:57.208+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:57.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:57.208+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:57.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:57.209+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:58.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:57.692+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:58.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:57.692+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:58.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:57.692+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:58.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:57.693+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:58.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:58.214+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:58.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:58.214+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:58.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:58.214+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:58.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:58.215+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:58.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:58.096+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:58.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:58.096+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:58.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:58.096+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:58.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:58.097+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:59.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:58.663+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:59.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:58.664+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:59.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:58.664+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:20:59.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:58.664+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:59.387 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:59.087+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:59.387 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:59.087+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:59.387 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:59.087+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:20:59.387 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:20:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:20:59.088+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:59.388 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:59.220+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:59.388 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:59.220+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:59.388 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:59.220+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:20:59.388 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:20:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:20:59.221+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:20:59.655 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:20:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:20:59.541+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 15822 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:21:00.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:59.652+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:00.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:59.652+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:00.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:59.652+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:00.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:20:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:20:59.652+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:00.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:00.057+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:00.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:00.057+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:00.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:00.057+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:00.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:00.058+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:00.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:00.219+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:00.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:00.219+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:00.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:00.219+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:00.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:00.220+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:01.014 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:00.665+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:01.014 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:00.665+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:01.015 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:00.665+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:01.015 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:00.666+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:01.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:01.011+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:01.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:01.011+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:01.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:01.011+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:01.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:01.012+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:01.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:01.187+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:01.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:01.187+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:01.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:01.187+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:01.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:01.187+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:02.014 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:01.623+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:02.015 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:01.623+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:02.015 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:01.623+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:02.015 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:01.625+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:02.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:02.011+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:02.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:02.011+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:02.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:02.011+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:02.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:02.012+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:02.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:02.226+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:02.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:02.226+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:02.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:02.226+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:02.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:02.227+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:03.018 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:02.635+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:03.019 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:02.635+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:03.019 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:02.635+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:03.019 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:02.636+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:03.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:03.195+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:03.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:03.195+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:03.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:03.195+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:03.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:03.195+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:03.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:03.015+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:03.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:03.015+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:03.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:03.015+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:03.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:03.016+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:04.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:03.622+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:04.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:03.622+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:04.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:03.622+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:04.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:03.624+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:04.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:04.039+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:04.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:04.039+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:04.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:04.039+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:04.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:04.040+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:04.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:04.171+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:04.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:04.171+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:04.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:04.171+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:04.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:04.171+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:05.029 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:21:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:21:04.543+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 15903 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:21:05.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:04.576+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:05.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:04.576+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:05.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:04.576+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:05.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:04.577+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:05.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:05.195+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:05.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:05.195+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:05.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:05.195+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:05.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:05.196+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:05.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:05.084+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:05.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:05.084+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:05.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:05.084+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:05.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:05.085+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:06.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:05.618+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:06.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:05.618+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:06.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:05.618+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:06.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:05.618+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:06.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:06.075+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:06.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:06.075+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:06.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:06.075+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:06.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:06.076+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:06.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:06.236+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:06.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:06.236+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:06.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:06.236+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:06.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:06.237+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:07.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:06.642+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:07.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:06.642+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:07.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:06.642+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:07.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:06.644+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:07.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:07.268+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:07.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:07.268+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:07.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:07.268+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:07.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:07.269+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:07.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:07.103+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:07.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:07.103+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:07.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:07.103+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:07.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:07.104+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:08.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:07.623+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:08.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:07.623+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:08.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:07.623+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:08.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:07.624+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:08.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:08.135+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:08.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:08.135+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:08.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:08.135+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:08.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:08.136+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:08.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:08.255+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:08.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:08.255+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:08.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:08.255+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:08.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:08.256+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:09.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:08.652+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:09.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:08.652+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:09.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:08.652+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:09.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:08.652+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:09.386 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:09.174+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:09.387 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:09.174+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:09.387 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:09.174+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:09.387 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:09.175+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:09.387 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:09.260+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:09.387 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:09.260+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:09.387 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:09.260+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:09.388 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:09.260+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:09.643 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:21:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:21:09.545+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 15982 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:21:10.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:09.640+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:10.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:09.640+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:10.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:09.640+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:10.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:09.641+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:10.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:10.217+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:10.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:10.217+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:10.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:10.217+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:10.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:10.218+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:10.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:10.306+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:10.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:10.306+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:10.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:10.306+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:10.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:10.307+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:11.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:10.681+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:11.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:10.681+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:11.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:10.681+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:11.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:10.682+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:11.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:11.335+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:11.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:11.335+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:11.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:11.335+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:11.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:11.336+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:11.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:11.198+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:11.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:11.198+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:11.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:11.198+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:11.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:11.199+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:12.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:11.643+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:12.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:11.643+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:12.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:11.643+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:12.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:11.644+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:12.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:12.355+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:12.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:12.355+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:12.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:12.355+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:12.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:12.356+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:12.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:12.188+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:12.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:12.188+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:12.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:12.188+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:12.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:12.189+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:13.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:12.634+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:13.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:12.634+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:13.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:12.634+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:13.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:12.635+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:13.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:13.203+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:13.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:13.203+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:13.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:13.203+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:13.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:13.204+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:13.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:13.371+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:13.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:13.371+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:13.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:13.371+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:13.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:13.371+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:14.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:13.586+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:14.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:13.586+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:14.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:13.586+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:14.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:13.587+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:14.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:14.246+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:14.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:14.246+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:14.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:14.246+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:14.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:14.247+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:14.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:14.346+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:14.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:14.346+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:14.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:14.346+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:14.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:14.347+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:15.029 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:21:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:21:14.547+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 16062 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:21:15.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:14.580+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:15.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:14.580+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:15.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:14.580+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:15.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:14.581+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:15.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:15.212+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:15.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:15.212+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:15.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:15.212+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:15.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:15.213+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:15.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:15.373+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:15.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:15.373+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:15.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:15.373+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:15.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:15.374+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:16.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:15.539+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:16.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:15.539+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:16.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:15.539+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:16.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:15.540+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:16.514 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:16.191+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:16.514 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:16.191+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:16.514 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:16.191+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:16.514 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:16.192+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:16.515 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:16.411+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:16.515 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:16.411+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:16.515 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:16.411+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:16.515 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:16.411+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:16.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:16.510+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:16.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:16.510+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:16.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:16.510+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:16.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:16.512+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:17.478 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:17.428+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:17.478 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:17.428+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:17.478 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:17.428+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:17.478 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:17.429+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:17.479 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:17.462+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:17.479 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:17.462+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:17.479 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:17.462+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:17.479 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:17.463+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:17.479 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:17.218+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:17.480 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:17.218+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:17.480 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:17.218+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:17.480 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:17.219+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:18.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:18.448+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:18.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:18.448+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:18.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:18.448+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:18.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:18.449+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:18.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:18.255+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:18.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:18.255+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:18.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:18.255+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:18.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:18.256+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:18.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:18.484+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:18.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:18.484+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:18.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:18.484+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:18.531 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:18.485+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:19.552 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:19.448+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:19.552 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:19.448+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:19.552 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:19.448+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:19.553 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:19.449+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:19.553 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:19.486+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:19.553 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:19.486+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:19.553 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:19.486+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:19.553 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:19.487+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:19.553 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:19.290+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:19.554 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:19.290+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:19.554 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:19.290+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:19.554 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:19.291+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:20.029 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:21:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:21:19.549+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 16141 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:21:20.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:20.267+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:20.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:20.267+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:20.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:20.267+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:20.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:20.268+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:20.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:20.429+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:20.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:20.429+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:20.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:20.429+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:20.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:20.430+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:20.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:20.487+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:20.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:20.487+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:20.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:20.487+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:20.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:20.487+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:21.701 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:21.299+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:21.702 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:21.299+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:21.702 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:21.299+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:21.702 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:21.300+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:21.702 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:21.535+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:21.703 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:21.535+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:21.703 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:21.535+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:21.703 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:21.535+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:21.703 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:21.450+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:21.703 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:21.450+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:21.703 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:21.450+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:21.703 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:21.451+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:22.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:22.297+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:22.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:22.297+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:22.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:22.297+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:22.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:22.298+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:22.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:22.450+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:22.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:22.450+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:22.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:22.450+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:22.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:22.451+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:22.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:22.522+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:22.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:22.522+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:22.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:22.522+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:22.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:22.523+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:23.621 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:23.521+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:23.621 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:23.521+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:23.622 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:23.521+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:23.622 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:23.522+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:23.622 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:23.348+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:23.622 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:23.348+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:23.622 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:23.348+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:23.622 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:23.349+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:23.623 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:23.481+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:23.623 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:23.481+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:23.623 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:23.481+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:23.623 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:23.482+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:24.676 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:21:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:21:24.550+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 16221 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:21:24.677 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:24.383+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:24.677 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:24.383+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:24.677 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:24.383+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:24.677 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:24.383+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:24.678 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:24.450+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:24.678 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:24.450+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:24.678 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:24.450+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:24.678 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:24.451+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:24.678 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:24.546+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:24.679 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:24.546+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:24.679 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:24.546+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:24.679 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:24.547+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:25.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:25.352+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:25.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:25.352+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:25.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:25.352+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:25.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:25.353+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:25.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:25.475+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:25.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:25.475+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:25.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:25.475+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:25.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:25.475+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:25.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:25.573+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:25.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:25.573+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:25.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:25.573+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:25.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:25.574+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:26.627 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:26.351+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:26.627 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:26.351+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:26.627 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:26.351+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:26.627 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:26.352+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:26.628 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:26.489+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:26.628 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:26.489+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:26.628 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:26.489+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:26.628 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:26.490+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:27.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:26.624+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:27.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:26.624+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:27.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:26.624+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:27.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:26.625+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:27.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:27.365+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:27.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:27.365+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:27.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:27.365+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:27.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:27.365+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:27.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:27.442+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:27.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:27.442+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:27.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:27.443+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:27.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:27.444+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:27.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:27.576+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:27.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:27.576+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:27.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:27.576+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:27.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:27.576+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:28.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:28.413+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:28.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:28.413+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:28.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:28.413+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:28.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:28.414+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:28.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:28.479+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:28.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:28.479+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:28.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:28.479+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:28.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:28.480+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:28.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:28.572+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:28.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:28.572+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:28.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:28.572+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:28.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:28.573+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:29.681 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:21:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:21:29.552+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 16301 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:21:29.681 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:29.523+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:29.681 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:29.523+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:29.681 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:29.523+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:29.681 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:29.524+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:29.682 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:29.436+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:29.682 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:29.436+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:29.682 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:29.436+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:29.682 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:29.436+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:29.682 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:29.444+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:29.683 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:29.444+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:29.683 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:29.444+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:29.683 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:29.445+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:30.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:30.450+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:30.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:30.450+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:30.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:30.450+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:30.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:30.450+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:30.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:30.529+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:30.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:30.529+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:30.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:30.529+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:30.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:30.530+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:30.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:30.452+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:30.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:30.452+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:30.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:30.452+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:30.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:30.452+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:31.703 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:31.446+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:31.705 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:31.446+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:31.705 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:31.446+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:31.705 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:31.446+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:31.705 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:31.447+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:31.706 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:31.447+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:31.706 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:31.447+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:31.706 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:31.448+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:31.706 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:31.531+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:31.706 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:31.531+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:31.707 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:31.531+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:31.707 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:31.532+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:32.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:32.409+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:32.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:32.409+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:32.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:32.409+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:32.780 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:32.409+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:32.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:32.473+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:32.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:32.473+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:32.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:32.473+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:32.781 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:32.474+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:32.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:32.541+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:32.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:32.541+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:32.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:32.541+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:32.782 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:32.542+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:33.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:33.518+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:33.782 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:33.518+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:33.782 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:33.518+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:33.782 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:33.518+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:33.782 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:33.428+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:33.782 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:33.428+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:33.783 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:33.428+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:33.783 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:33.429+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:33.783 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:33.449+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:33.783 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:33.449+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:33.783 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:33.449+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:33.783 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:33.450+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:34.559 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:34.489+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:34.559 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:34.489+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:34.560 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:34.489+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:34.560 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:34.490+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:34.560 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:34.429+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:34.560 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:34.429+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:34.560 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:34.429+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:34.560 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:34.430+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:35.029 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:21:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:21:34.555+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 16381 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:21:35.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:34.553+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:35.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:34.553+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:35.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:34.553+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:35.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:34.554+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:35.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:35.424+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:35.783 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:35.424+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:35.783 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:35.424+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:35.783 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:35.425+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:35.784 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:35.446+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:35.784 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:35.446+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:35.784 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:35.446+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:35.784 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:35.446+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:35.784 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:35.590+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:35.785 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:35.590+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:35.785 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:35.590+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:35.785 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:35.591+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:36.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:36.461+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:36.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:36.461+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:36.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:36.461+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:36.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:36.462+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:36.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:36.463+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:36.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:36.463+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:36.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:36.463+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:36.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:36.464+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:36.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:36.593+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:36.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:36.593+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:36.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:36.593+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:36.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:36.593+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:37.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:37.473+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:37.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:37.473+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:37.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:37.473+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:37.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:37.474+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:37.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:37.471+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:37.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:37.471+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:37.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:37.471+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:37.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:37.472+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:37.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:37.594+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:37.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:37.594+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:37.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:37.594+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:37.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:37.595+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:38.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:38.443+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:38.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:38.443+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:38.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:38.443+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:38.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:38.443+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:38.779 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:38.599+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:38.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:38.599+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:38.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:38.599+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:38.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:38.600+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:38.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:38.474+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:38.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:38.474+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:38.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:38.474+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:38.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:38.475+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:39.385 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-11T23:21:39.386 DEBUG:tasks.cephfs.mount:Force/lazy unmounting on client.0 2024-03-11T23:21:39.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:21:39.387 DEBUG:teuthology.orchestra.run.smithi153:> sudo umount --lazy --force /home/ubuntu/cephtest/mnt.0 2024-03-11T23:21:39.427 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/io_destroy: Cannot mkdir: Input/output error 2024-03-11T23:21:39.428 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/io_destroy/io_destroy01.c: Cannot open: Input/output error 2024-03-11T23:21:39.428 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/io_destroy/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.428 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/ustat: Cannot mkdir: Input/output error 2024-03-11T23:21:39.428 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/ustat/ustat02.c: Cannot open: Input/output error 2024-03-11T23:21:39.428 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/ustat/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.428 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/ustat/ustat01.c: Cannot open: Input/output error 2024-03-11T23:21:39.428 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigsuspend: Cannot mkdir: Input/output error 2024-03-11T23:21:39.429 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigsuspend/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.429 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigsuspend/sigsuspend01.c: Cannot open: Input/output error 2024-03-11T23:21:39.429 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mq_unlink: Cannot mkdir: Input/output error 2024-03-11T23:21:39.429 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mq_unlink/mq_unlink01.c: Cannot open: Input/output error 2024-03-11T23:21:39.429 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mq_unlink/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.429 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getcpu: Cannot mkdir: Input/output error 2024-03-11T23:21:39.429 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getcpu/getcpu01.c: Cannot open: Input/output error 2024-03-11T23:21:39.429 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getcpu/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.429 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/pipe2: Cannot mkdir: Input/output error 2024-03-11T23:21:39.430 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/pipe2/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.430 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/pipe2/pipe2_01.c: Cannot open: Input/output error 2024-03-11T23:21:39.430 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/pipe2/pipe2_02.c: Cannot open: Input/output error 2024-03-11T23:21:39.430 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mkdir: Cannot mkdir: Input/output error 2024-03-11T23:21:39.430 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mkdir/mkdir09.c: Cannot open: Input/output error 2024-03-11T23:21:39.430 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mkdir/mkdir08.c: Cannot open: Input/output error 2024-03-11T23:21:39.430 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mkdir/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.430 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mkdir/mkdir01.c: Cannot open: Input/output error 2024-03-11T23:21:39.430 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mkdir/mkdir04.c: Cannot open: Input/output error 2024-03-11T23:21:39.431 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mkdir/mkdir05.c: Cannot open: Input/output error 2024-03-11T23:21:39.431 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mkdir/mkdir02.c: Cannot open: Input/output error 2024-03-11T23:21:39.431 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mkdir/mkdir03.c: Cannot open: Input/output error 2024-03-11T23:21:39.431 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/uname: Cannot mkdir: Input/output error 2024-03-11T23:21:39.431 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/uname/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.431 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/uname/uname01.c: Cannot open: Input/output error 2024-03-11T23:21:39.431 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/uname/uname03.c: Cannot open: Input/output error 2024-03-11T23:21:39.431 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/uname/uname02.c: Cannot open: Input/output error 2024-03-11T23:21:39.431 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getegid: Cannot mkdir: Input/output error 2024-03-11T23:21:39.432 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getegid/getegid02.c: Cannot open: Input/output error 2024-03-11T23:21:39.432 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getegid/compat_16.h: Cannot open: Input/output error 2024-03-11T23:21:39.432 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getegid/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.432 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getegid/getegid01.c: Cannot open: Input/output error 2024-03-11T23:21:39.432 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/read: Cannot mkdir: Input/output error 2024-03-11T23:21:39.432 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/read/read03.c: Cannot open: Input/output error 2024-03-11T23:21:39.432 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/read/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.432 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/read/read02.c: Cannot open: Input/output error 2024-03-11T23:21:39.433 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/read/read04.c: Cannot open: Input/output error 2024-03-11T23:21:39.433 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/read/read01.c: Cannot open: Input/output error 2024-03-11T23:21:39.433 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mlock: Cannot mkdir: Input/output error 2024-03-11T23:21:39.433 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mlock/mlock01.c: Cannot open: Input/output error 2024-03-11T23:21:39.433 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mlock/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.433 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mlock/mlock02.c: Cannot open: Input/output error 2024-03-11T23:21:39.433 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/poll: Cannot mkdir: Input/output error 2024-03-11T23:21:39.433 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/poll/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.433 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/poll/poll01.c: Cannot open: Input/output error 2024-03-11T23:21:39.434 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/io_cancel: Cannot mkdir: Input/output error 2024-03-11T23:21:39.434 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/io_cancel/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.434 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/io_cancel/io_cancel01.c: Cannot open: Input/output error 2024-03-11T23:21:39.434 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/execlp: Cannot mkdir: Input/output error 2024-03-11T23:21:39.434 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/execlp/execlp01.c: Cannot open: Input/output error 2024-03-11T23:21:39.434 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/execlp/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.434 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/stat: Cannot mkdir: Input/output error 2024-03-11T23:21:39.434 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/stat/stat06.c: Cannot open: Input/output error 2024-03-11T23:21:39.434 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/stat/stat02.c: Cannot open: Input/output error 2024-03-11T23:21:39.435 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/stat/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.435 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/stat/stat03.c: Cannot open: Input/output error 2024-03-11T23:21:39.435 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/stat/stat05.c: Cannot open: Input/output error 2024-03-11T23:21:39.435 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/stat/stat01.c: Cannot open: Input/output error 2024-03-11T23:21:39.435 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fstatfs: Cannot mkdir: Input/output error 2024-03-11T23:21:39.435 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fstatfs/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.435 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fstatfs/fstatfs01.c: Cannot open: Input/output error 2024-03-11T23:21:39.435 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/fstatfs/fstatfs02.c: Cannot open: Input/output error 2024-03-11T23:21:39.436 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clock_nanosleep: Cannot mkdir: Input/output error 2024-03-11T23:21:39.436 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clock_nanosleep/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.436 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clock_nanosleep/clock_nanosleep01.c: Cannot open: Input/output error 2024-03-11T23:21:39.436 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/symlink: Cannot mkdir: Input/output error 2024-03-11T23:21:39.436 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/symlink/symlink02.c: Cannot open: Input/output error 2024-03-11T23:21:39.436 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/symlink/symlink01.c: Cannot open: Input/output error 2024-03-11T23:21:39.436 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/symlink/symlink05.c: Cannot open: Input/output error 2024-03-11T23:21:39.436 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/symlink/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.437 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/symlink/symlink03.c: Cannot open: Input/output error 2024-03-11T23:21:39.437 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/symlink/symlink04.c: Cannot open: Input/output error 2024-03-11T23:21:39.437 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mprotect: Cannot mkdir: Input/output error 2024-03-11T23:21:39.437 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mprotect/mprotect01.c: Cannot open: Input/output error 2024-03-11T23:21:39.437 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mprotect/mprotect02.c: Cannot open: Input/output error 2024-03-11T23:21:39.437 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mprotect/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.437 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mprotect/mprotect03.c: Cannot open: Input/output error 2024-03-11T23:21:39.437 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rmdir: Cannot mkdir: Input/output error 2024-03-11T23:21:39.437 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rmdir/rmdir02.c: Cannot open: Input/output error 2024-03-11T23:21:39.437 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rmdir/rmdir04.c: Cannot open: Input/output error 2024-03-11T23:21:39.438 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rmdir/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.439 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rmdir/rmdir03.c: Cannot open: Input/output error 2024-03-11T23:21:39.439 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rmdir/rmdir05.c: Cannot open: Input/output error 2024-03-11T23:21:39.439 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/rmdir/rmdir01.c: Cannot open: Input/output error 2024-03-11T23:21:39.439 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigaction: Cannot mkdir: Input/output error 2024-03-11T23:21:39.440 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigaction/sigaction01.c: Cannot open: Input/output error 2024-03-11T23:21:39.440 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigaction/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.440 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sigaction/sigaction02.c: Cannot open: Input/output error 2024-03-11T23:21:39.440 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw: Cannot mkdir: Input/output error 2024-03-11T23:21:39.440 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/lib.c: Cannot open: Input/output error 2024-03-11T23:21:39.440 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/nftw.h: Cannot open: Input/output error 2024-03-11T23:21:39.440 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/lib64.c: Cannot open: Input/output error 2024-03-11T23:21:39.440 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/test.c: Cannot open: Input/output error 2024-03-11T23:21:39.440 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/test_func.c: Cannot open: Input/output error 2024-03-11T23:21:39.441 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/test_func64.c: Cannot open: Input/output error 2024-03-11T23:21:39.441 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.441 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/nftw64.c: Cannot open: Input/output error 2024-03-11T23:21:39.441 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/test64.c: Cannot open: Input/output error 2024-03-11T23:21:39.441 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/nftw64.h: Cannot open: Input/output error 2024-03-11T23:21:39.441 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/tools.c: Cannot open: Input/output error 2024-03-11T23:21:39.441 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/nftw.c: Cannot open: Input/output error 2024-03-11T23:21:39.441 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nftw/tools64.c: Cannot open: Input/output error 2024-03-11T23:21:39.441 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mq_timedsend: Cannot mkdir: Input/output error 2024-03-11T23:21:39.442 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mq_timedsend/mq_timedsend01.c: Cannot open: Input/output error 2024-03-11T23:21:39.442 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mq_timedsend/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.442 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/statvfs: Cannot mkdir: Input/output error 2024-03-11T23:21:39.442 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/statvfs/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.442 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/statvfs/statvfs01.c: Cannot open: Input/output error 2024-03-11T23:21:39.442 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/reboot: Cannot mkdir: Input/output error 2024-03-11T23:21:39.442 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/reboot/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.442 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/reboot/reboot01.c: Cannot open: Input/output error 2024-03-11T23:21:39.442 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/reboot/reboot02.c: Cannot open: Input/output error 2024-03-11T23:21:39.443 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/confstr: Cannot mkdir: Input/output error 2024-03-11T23:21:39.443 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/confstr/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.443 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/confstr/confstr01.c: Cannot open: Input/output error 2024-03-11T23:21:39.443 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sysinfo: Cannot mkdir: Input/output error 2024-03-11T23:21:39.443 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sysinfo/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.443 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sysinfo/sysinfo02.c: Cannot open: Input/output error 2024-03-11T23:21:39.443 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sysinfo/sysinfo01.c: Cannot open: Input/output error 2024-03-11T23:21:39.443 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/execv: Cannot mkdir: Input/output error 2024-03-11T23:21:39.443 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/execv/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.444 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/execv/execv01.c: Cannot open: Input/output error 2024-03-11T23:21:39.444 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/cacheflush: Cannot mkdir: Input/output error 2024-03-11T23:21:39.444 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/cacheflush/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.444 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/cacheflush/cacheflush01.c: Cannot open: Input/output error 2024-03-11T23:21:39.444 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nanosleep: Cannot mkdir: Input/output error 2024-03-11T23:21:39.444 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nanosleep/nanosleep01.c: Cannot open: Input/output error 2024-03-11T23:21:39.444 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nanosleep/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.444 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nanosleep/nanosleep04.c: Cannot open: Input/output error 2024-03-11T23:21:39.444 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nanosleep/nanosleep03.c: Cannot open: Input/output error 2024-03-11T23:21:39.445 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/nanosleep/nanosleep02.c: Cannot open: Input/output error 2024-03-11T23:21:39.445 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mq_open: Cannot mkdir: Input/output error 2024-03-11T23:21:39.445 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mq_open/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.445 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/mq_open/mq_open01.c: Cannot open: Input/output error 2024-03-11T23:21:39.445 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getgid: Cannot mkdir: Input/output error 2024-03-11T23:21:39.445 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getgid/compat_16.h: Cannot open: Input/output error 2024-03-11T23:21:39.445 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getgid/getgid03.c: Cannot open: Input/output error 2024-03-11T23:21:39.445 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getgid/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.445 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/getgid/getgid01.c: Cannot open: Input/output error 2024-03-11T23:21:39.446 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/vhangup: Cannot mkdir: Input/output error 2024-03-11T23:21:39.446 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/vhangup/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.446 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/vhangup/vhangup02.c: Cannot open: Input/output error 2024-03-11T23:21:39.446 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/vhangup/vhangup01.c: Cannot open: Input/output error 2024-03-11T23:21:39.446 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/gethostname: Cannot mkdir: Input/output error 2024-03-11T23:21:39.446 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/gethostname/gethostname01.c: Cannot open: Input/output error 2024-03-11T23:21:39.446 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/gethostname/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.446 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/acct: Cannot mkdir: Input/output error 2024-03-11T23:21:39.446 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/acct/acct02.c: Cannot open: Input/output error 2024-03-11T23:21:39.447 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/acct/acct01.c: Cannot open: Input/output error 2024-03-11T23:21:39.447 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/acct/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.447 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_getscheduler: Cannot mkdir: Input/output error 2024-03-11T23:21:39.447 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_getscheduler/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.447 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_getscheduler/sched_getscheduler02.c: Cannot open: Input/output error 2024-03-11T23:21:39.447 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/sched_getscheduler/sched_getscheduler01.c: Cannot open: Input/output error 2024-03-11T23:21:39.447 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clone: Cannot mkdir: Input/output error 2024-03-11T23:21:39.447 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clone/clone02.c: Cannot open: Input/output error 2024-03-11T23:21:39.447 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clone/clone_platform.h: Cannot open: Input/output error 2024-03-11T23:21:39.448 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clone/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.448 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clone/clone04.c: Cannot open: Input/output error 2024-03-11T23:21:39.448 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clone/clone03.c: Cannot open: Input/output error 2024-03-11T23:21:39.448 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clone/clone06.c: Cannot open: Input/output error 2024-03-11T23:21:39.448 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clone/clone07.c: Cannot open: Input/output error 2024-03-11T23:21:39.448 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clone/clone05.c: Cannot open: Input/output error 2024-03-11T23:21:39.448 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/clone/clone01.c: Cannot open: Input/output error 2024-03-11T23:21:39.448 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setresgid: Cannot mkdir: Input/output error 2024-03-11T23:21:39.448 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setresgid/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.449 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setresgid/setresgid01.c: Cannot open: Input/output error 2024-03-11T23:21:39.449 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setresgid/setresgid02.c: Cannot open: Input/output error 2024-03-11T23:21:39.449 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setresgid/setresgid03.c: Cannot open: Input/output error 2024-03-11T23:21:39.449 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/brk: Cannot mkdir: Input/output error 2024-03-11T23:21:39.449 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/brk/brk01.c: Cannot open: Input/output error 2024-03-11T23:21:39.449 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/brk/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.449 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setreuid: Cannot mkdir: Input/output error 2024-03-11T23:21:39.449 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setreuid/setreuid03.c: Cannot open: Input/output error 2024-03-11T23:21:39.449 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setreuid/Makefile: Cannot open: Input/output error 2024-03-11T23:21:39.449 INFO:tasks.workunit.client.0.smithi153.stderr: 2024-03-11T23:21:39.450 INFO:tasks.workunit.client.0.smithi153.stderr:gzip: stdin: Stale file handle 2024-03-11T23:21:39.450 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls/setreuid/setreuid07.c: Cannot open: Input/output error 2024-03-11T23:21:39.450 INFO:tasks.workunit.client.0.smithi153.stderr:tar: Unexpected EOF in archive 2024-03-11T23:21:39.450 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel/syscalls: Cannot utime: Input/output error 2024-03-11T23:21:39.451 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases/kernel: Cannot utime: Input/output error 2024-03-11T23:21:39.451 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231/testcases: Cannot utime: Input/output error 2024-03-11T23:21:39.451 INFO:tasks.workunit.client.0.smithi153.stderr:tar: ltp-full-20091231: Cannot utime: Input/output error 2024-03-11T23:21:39.451 INFO:tasks.workunit.client.0.smithi153.stderr:tar: Error is not recoverable: exiting now 2024-03-11T23:21:39.451 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi153.front.sepia.ceph.com 2024-03-11T23:21:39.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-11T23:21:39.452 DEBUG:teuthology.orchestra.run.smithi153:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-11T23:21:39.452 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-03-11T23:21:39.453 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-03-11T23:21:39.453 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-11T23:21:39.528 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-11T23:21:39.528 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-11T23:21:39.688 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:21:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:21:39.556+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 16461 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:21:39.688 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:39.518+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:39.688 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:39.518+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:39.688 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:39.518+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:39.689 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:39.520+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:39.689 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:39.417+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:39.689 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:39.417+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:39.689 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:39.417+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:39.689 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:39.418+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:39.690 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:39.576+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:39.690 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:39.576+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:39.690 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:39.576+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:39.690 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:39.577+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:40.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:40.402+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:40.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:40.402+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:40.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:40.402+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:40.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:40.403+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:40.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:40.527+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:40.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:40.527+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:40.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:40.527+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:40.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:40.528+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:40.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:40.609+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:40.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:40.609+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:40.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:40.609+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:40.781 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:40.610+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:41.703 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:41.519+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:41.703 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:41.519+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:41.703 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:41.519+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:41.703 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:41.520+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:41.704 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:41.605+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:41.704 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:41.605+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:41.704 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:41.605+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:41.704 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:41.606+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:41.704 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:41.371+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:41.704 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:41.371+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:41.704 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:41.371+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:41.705 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:41.372+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:42.634 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:42.552+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:42.634 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:42.552+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:42.634 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:42.552+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:42.634 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:42.553+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:42.635 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:42.343+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:42.635 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:42.343+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:42.635 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:42.343+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:42.635 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:42.344+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:43.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:42.631+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:43.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:42.631+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:43.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:42.631+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:43.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:42.632+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:43.588 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:43.306+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:43.588 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:43.306+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:43.588 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:43.306+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:43.588 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:43.307+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:43.589 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:43.505+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:43.589 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:43.505+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:43.589 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:43.505+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:43.589 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:43.506+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:44.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:43.669+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:44.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:43.669+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:44.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:43.669+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:44.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:43.670+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:44.572 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:44.310+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:44.572 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:44.310+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:44.573 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:44.310+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:44.573 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:44.311+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:44.573 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:44.536+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:44.573 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:44.536+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:44.573 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:44.536+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:44.574 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:44.536+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:44.574 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:21:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:21:44.558+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 16541 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:21:45.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:44.668+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:45.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:44.668+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:45.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:44.668+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:45.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:44.669+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:45.490 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:45.305+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:45.491 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:45.305+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:45.491 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:45.305+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:45.491 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:45.306+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:45.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:45.488+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:45.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:45.488+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:45.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:45.488+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:45.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:45.488+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:45.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:45.671+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:45.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:45.671+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:45.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:45.671+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:45.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:45.672+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:46.539 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:46.281+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:46.539 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:46.281+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:46.539 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:46.281+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:46.539 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:46.282+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:47.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:46.536+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:47.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:46.536+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:47.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:46.536+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:47.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:46.536+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:47.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:46.677+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:47.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:46.677+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:47.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:46.677+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:47.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:46.678+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:47.510 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:47.247+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:47.511 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:47.247+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:47.511 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:47.247+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:47.511 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:47.248+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:47.778 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:47.710+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:47.779 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:47.710+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:47.779 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:47.710+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:47.779 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:47.711+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:47.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:47.508+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:47.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:47.508+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:47.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:47.508+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:47.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:47.508+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:48.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:48.273+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:48.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:48.273+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:48.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:48.273+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:48.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:48.274+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:48.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:48.467+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:48.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:48.467+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:48.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:48.467+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:48.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:48.467+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:49.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:48.752+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:49.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:48.752+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:49.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:48.752+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:49.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:48.752+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:49.504 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:49.251+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:49.504 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:49.251+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:49.504 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:49.251+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:49.504 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:49.252+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:49.778 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:21:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:21:49.560+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 16621 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:21:49.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:49.501+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:49.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:49.501+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:49.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:49.501+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:49.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:49.502+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:49.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:49.735+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:49.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:49.735+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:49.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:49.735+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:49.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:49.736+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:50.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:50.469+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:50.529 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:50.469+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:50.529 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:50.469+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:50.529 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:50.470+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:50.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:50.254+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:50.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:50.254+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:50.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:50.254+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:50.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:50.255+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:51.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:50.754+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:51.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:50.754+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:51.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:50.754+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:51.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:50.754+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:51.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:51.263+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:51.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:51.263+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:51.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:51.263+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:51.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:51.264+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:51.529 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:51.492+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:51.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:51.492+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:51.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:51.492+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:51.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:51.492+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:52.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:51.798+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:52.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:51.798+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:52.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:51.798+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:52.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:51.799+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:52.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:52.281+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:52.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:52.281+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:52.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:52.281+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:52.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:52.282+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:52.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:52.466+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:52.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:52.466+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:52.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:52.466+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:52.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:52.467+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:53.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:52.843+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:53.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:52.843+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:53.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:52.843+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:53.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:52.844+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:53.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:53.272+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:53.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:53.272+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:53.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:53.272+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:53.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:53.273+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:53.529 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:53.479+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:53.529 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:53.479+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:53.529 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:53.479+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:53.530 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:53.480+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:54.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:53.842+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:54.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:53.842+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:54.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:53.842+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:54.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:53.843+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:54.565 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:54.487+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:54.565 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:54.487+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:54.566 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:54.487+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:54.566 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:54.487+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:54.566 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:54.297+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:54.566 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:54.297+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:54.566 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:54.297+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:54.566 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:54.298+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:54.849 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:21:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:21:54.562+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 16701 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:21:55.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:54.846+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:55.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:54.846+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:55.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:54.846+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:55.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:54.847+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:55.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:55.468+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:55.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:55.468+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:55.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:55.468+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:55.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:55.468+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:55.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:55.280+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:55.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:55.280+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:55.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:55.280+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:55.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:55.281+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:56.259 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:55.847+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:56.259 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:55.847+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:56.259 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:55.847+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:56.260 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:55.848+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:56.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:56.502+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:56.529 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:56.502+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:56.529 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:56.502+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:56.529 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:56.503+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:56.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:56.256+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:56.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:56.256+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:56.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:56.256+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:56.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:56.257+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:57.244 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:56.824+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:57.244 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:56.824+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:57.244 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:56.824+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:57.245 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:56.825+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:57.529 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:57.477+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:57.529 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:57.477+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:57.529 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:57.477+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:57.529 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:57.478+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:57.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:57.241+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:57.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:57.241+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:57.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:57.241+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:57.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:57.242+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:58.256 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:57.832+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:58.256 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:57.832+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:58.256 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:57.832+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:58.256 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:57.833+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:58.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:58.253+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:58.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:58.253+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:58.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:58.253+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:58.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:58.254+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:58.529 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:58.460+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:58.529 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:58.460+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:58.529 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:58.460+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:58.529 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:58.461+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:59.228 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:58.840+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:59.228 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:58.840+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:59.228 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:58.840+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:21:59.229 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:58.841+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:59.501 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:59.225+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:59.501 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:59.225+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:59.501 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:59.225+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:21:59.502 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:21:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:21:59.226+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:21:59.778 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:21:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:21:59.564+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 16781 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:21:59.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:59.498+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:59.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:59.498+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:59.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:59.498+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:21:59.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:21:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:21:59.499+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:00.210 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:59.850+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:00.211 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:59.850+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:00.211 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:59.850+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:00.211 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:21:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:21:59.851+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:00.490 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:00.208+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:00.490 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:00.208+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:00.490 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:00.208+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:00.490 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:00.209+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:00.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:00.487+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:00.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:00.487+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:00.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:00.487+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:00.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:00.488+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:01.170 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:00.895+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:01.170 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:00.895+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:01.170 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:00.895+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:01.170 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:00.896+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:01.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:01.167+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:01.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:01.167+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:01.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:01.167+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:01.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:01.168+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:01.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:01.525+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:01.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:01.525+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:01.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:01.525+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:01.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:01.525+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:02.210 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:01.912+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:02.211 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:01.912+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:02.211 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:01.912+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:02.211 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:01.913+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:02.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:02.208+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:02.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:02.208+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:02.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:02.208+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:02.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:02.208+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:02.921 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:02.569+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:02.921 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:02.569+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:02.921 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:02.569+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:02.921 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:02.570+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:03.198 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:02.918+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:03.199 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:02.918+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:03.199 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:02.918+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:03.199 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:02.919+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:03.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:03.196+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:03.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:03.196+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:03.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:03.196+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:03.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:03.196+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:03.885 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:03.548+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:03.885 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:03.548+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:03.886 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:03.548+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:03.886 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:03.549+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:04.231 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:03.882+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:04.232 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:03.882+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:04.232 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:03.882+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:04.232 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:03.884+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:04.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:04.229+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:04.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:04.229+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:04.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:04.229+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:04.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:04.229+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:04.859 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:22:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:22:04.566+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 16861 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:22:04.860 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:04.552+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:04.860 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:04.552+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:04.860 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:04.552+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:04.860 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:04.553+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:05.222 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:04.857+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:05.223 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:04.857+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:05.223 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:04.857+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:05.223 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:04.858+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:05.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:05.220+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:05.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:05.220+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:05.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:05.220+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:05.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:05.221+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:05.833 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:05.570+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:05.834 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:05.570+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:05.834 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:05.570+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:05.834 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:05.572+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:06.257 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:05.831+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:06.257 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:05.831+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:06.257 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:05.831+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:06.257 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:05.832+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:06.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:06.254+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:06.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:06.254+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:06.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:06.254+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:06.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:06.255+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:06.866 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:06.598+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:06.866 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:06.598+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:06.866 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:06.598+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:06.866 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:06.598+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:07.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:06.863+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:07.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:06.863+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:07.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:06.863+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:07.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:06.864+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:07.552 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:07.297+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:07.552 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:07.297+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:07.552 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:07.297+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:07.552 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:07.297+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:07.838 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:07.550+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:07.839 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:07.550+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:07.839 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:07.550+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:07.839 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:07.551+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:08.274 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:07.836+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:08.274 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:07.836+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:08.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:07.836+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:08.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:07.837+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:08.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:08.271+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:08.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:08.271+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:08.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:08.271+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:08.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:08.272+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:08.855 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:08.578+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:08.856 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:08.578+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:08.856 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:08.578+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:08.856 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:08.579+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:09.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:08.853+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:09.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:08.853+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:09.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:08.853+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:09.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:08.853+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:09.571 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:09.304+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:09.571 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:09.304+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:09.571 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:09.304+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:09.571 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:09.304+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:09.865 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:22:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:22:09.568+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 16941 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:22:09.866 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:09.585+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:09.866 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:09.585+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:09.866 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:09.585+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:09.866 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:09.585+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:10.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:09.863+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:10.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:09.863+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:10.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:09.863+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:10.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:09.864+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:10.610 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:10.349+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:10.610 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:10.349+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:10.610 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:10.349+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:10.611 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:10.350+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:10.900 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:10.607+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:10.900 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:10.607+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:10.901 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:10.607+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:10.901 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:10.608+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:11.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:10.898+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:11.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:10.898+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:11.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:10.898+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:11.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:10.898+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:11.704 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:11.325+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:11.704 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:11.325+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:11.704 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:11.325+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:11.704 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:11.326+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:11.705 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:11.561+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:11.705 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:11.561+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:11.705 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:11.561+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:11.705 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:11.562+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:12.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:11.934+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:12.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:11.934+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:12.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:11.934+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:12.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:11.934+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:12.614 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:12.352+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:12.615 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:12.352+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:12.615 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:12.352+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:12.615 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:12.353+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:12.887 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:12.612+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:12.888 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:12.612+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:12.888 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:12.612+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:12.888 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:12.613+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:13.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:12.885+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:13.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:12.885+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:13.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:12.885+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:13.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:12.886+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:13.590 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:13.325+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:13.590 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:13.325+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:13.590 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:13.325+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:13.590 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:13.326+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:13.862 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:13.628+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:13.862 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:13.629+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:13.862 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:13.629+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:13.862 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:13.629+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:14.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:13.859+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:14.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:13.859+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:14.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:13.859+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:14.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:13.860+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:14.621 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:22:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:22:14.570+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 17021 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:22:14.621 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:14.355+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:14.621 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:14.355+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:14.622 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:14.355+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:14.622 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:14.356+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:15.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:14.840+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:15.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:14.840+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:15.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:14.840+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:15.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:14.842+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:15.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:14.618+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:15.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:14.618+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:15.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:14.618+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:15.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:14.619+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:15.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:15.373+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:15.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:15.373+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:15.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:15.373+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:15.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:15.374+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:15.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:15.602+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:15.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:15.602+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:15.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:15.602+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:15.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:15.603+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:16.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:15.850+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:16.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:15.850+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:16.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:15.850+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:16.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:15.850+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:16.603 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:16.334+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:16.603 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:16.334+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:16.603 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:16.334+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:16.603 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:16.335+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:17.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:16.600+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:17.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:16.600+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:17.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:16.600+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:17.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:16.601+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:17.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:16.841+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:17.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:16.841+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:17.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:16.841+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:17.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:16.842+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:17.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:17.316+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:17.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:17.316+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:17.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:17.316+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:17.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:17.316+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:17.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:17.561+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:17.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:17.561+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:17.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:17.561+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:17.780 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:17.561+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:18.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:17.828+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:18.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:17.828+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:18.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:17.828+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:18.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:17.829+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:18.580 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:18.304+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:18.580 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:18.304+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:18.580 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:18.304+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:18.580 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:18.305+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:18.840 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:18.577+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:18.841 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:18.577+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:18.841 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:18.577+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:18.841 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:18.578+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:19.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:18.838+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:19.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:18.838+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:19.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:18.838+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:19.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:18.838+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:19.574 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:19.320+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:19.575 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:19.320+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:19.575 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:19.320+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:19.575 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:19.321+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:19.841 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:19.602+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:19.842 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:19.602+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:19.842 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:19.602+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:19.842 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:19.603+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:19.842 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:22:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:22:19.571+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 17101 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:22:20.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:19.838+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:20.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:19.839+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:20.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:19.839+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:20.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:19.840+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:20.580 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:20.298+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:20.580 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:20.298+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:20.581 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:20.298+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:20.581 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:20.299+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:20.869 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:20.578+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:20.870 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:20.578+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:20.870 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:20.578+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:20.870 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:20.578+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:21.258 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:20.866+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:21.258 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:20.866+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:21.259 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:20.866+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:21.259 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:20.867+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:21.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:21.256+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:21.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:21.256+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:21.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:21.256+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:21.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:21.256+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:21.916 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:21.609+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:21.916 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:21.609+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:21.916 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:21.609+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:21.916 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:21.610+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:22.239 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:21.913+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:22.239 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:21.913+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:22.239 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:21.913+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:22.239 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:21.914+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:22.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:22.236+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:22.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:22.236+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:22.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:22.236+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:22.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:22.237+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:22.945 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:22.560+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:22.946 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:22.560+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:22.946 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:22.560+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:22.946 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:22.560+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:23.204 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:22.943+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:23.204 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:22.943+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:23.204 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:22.943+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:23.204 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:22.944+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:23.467 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:23.201+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:23.467 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:23.201+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:23.467 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:23.201+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:23.467 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:23.202+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:23.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:23.586+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:23.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:23.586+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:23.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:23.586+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:23.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:23.587+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:24.139 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:23.930+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:24.140 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:23.930+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:24.140 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:23.930+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:24.140 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:23.931+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:24.392 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:24.222+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:24.393 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:24.222+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:24.393 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:24.222+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:24.393 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:24.222+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:24.699 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:22:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:22:24.573+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 17181 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:22:24.699 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:24.606+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:24.699 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:24.606+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:24.699 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:24.606+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:24.700 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:24.607+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:25.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:24.921+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:25.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:24.921+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:25.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:24.921+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:25.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:24.922+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:25.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:25.231+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:25.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:25.231+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:25.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:25.231+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:25.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:25.232+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:25.884 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:25.598+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:25.884 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:25.598+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:25.884 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:25.598+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:25.885 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:25.599+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:26.230 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:25.881+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:26.230 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:25.881+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:26.231 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:25.881+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:26.231 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:25.882+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:26.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:26.227+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:26.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:26.228+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:26.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:26.228+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:26.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:26.229+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:26.911 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:26.619+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:26.912 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:26.619+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:26.912 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:26.619+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:26.912 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:26.620+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:27.196 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:26.908+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:27.196 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:26.908+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:27.196 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:26.908+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:27.196 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:26.909+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:27.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:27.193+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:27.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:27.193+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:27.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:27.193+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:27.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:27.194+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:27.916 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:27.588+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:27.916 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:27.588+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:27.916 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:27.588+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:27.916 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:27.589+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:28.171 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:27.913+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:28.172 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:27.913+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:28.172 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:27.913+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:28.172 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:27.914+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:28.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:28.169+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:28.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:28.169+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:28.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:28.169+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:28.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:28.170+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:28.887 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:28.590+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:28.887 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:28.590+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:28.887 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:28.590+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:28.887 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:28.591+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:29.188 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:28.884+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:29.188 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:28.884+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:29.188 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:28.884+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:29.188 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:28.885+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:29.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:29.185+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:29.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:29.185+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:29.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:29.185+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:29.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:29.186+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:29.931 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:22:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:22:29.575+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 17261 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:22:29.931 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:29.574+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:29.932 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:29.574+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:29.932 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:29.574+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:29.932 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:29.575+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:30.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:30.153+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:30.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:30.153+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:30.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:30.153+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:30.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:30.154+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:30.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:29.928+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:30.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:29.928+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:30.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:29.928+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:30.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:29.930+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:30.969 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:30.580+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:30.969 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:30.580+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:30.969 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:30.580+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:30.970 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:30.581+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:31.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:30.966+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:31.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:30.966+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:31.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:30.966+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:31.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:30.967+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:31.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:31.151+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:31.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:31.151+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:31.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:31.151+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:31.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:31.151+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:31.995 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:31.610+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:31.996 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:31.610+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:31.996 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:31.610+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:31.996 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:31.611+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:32.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:32.151+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:32.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:32.151+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:32.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:32.151+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:32.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:32.151+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:32.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:31.993+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:32.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:31.993+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:32.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:31.993+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:32.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:31.994+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:33.001 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:32.624+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:33.001 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:32.624+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:33.001 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:32.624+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:33.001 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:32.625+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:33.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:33.147+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:33.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:33.147+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:33.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:33.147+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:33.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:33.148+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:33.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:32.998+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:33.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:32.998+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:33.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:32.998+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:33.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:32.999+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:33.990 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:33.988+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:33.991 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:33.988+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:33.991 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:33.988+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:33.991 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:33.666+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:33.991 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:33.666+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:33.991 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:33.666+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:33.991 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:33.667+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:34.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:33.988+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:34.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:34.119+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:34.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:34.119+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:34.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:34.119+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:34.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:34.120+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:34.803 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:22:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:22:34.577+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 17341 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:22:34.804 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:34.676+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:34.804 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:34.676+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:34.804 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:34.676+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:34.804 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:34.677+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:35.104 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:35.020+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:35.105 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:35.020+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:35.105 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:35.020+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:35.105 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:35.021+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:35.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:35.102+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:35.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:35.102+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:35.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:35.102+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:35.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:35.103+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:36.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:35.628+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:36.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:35.628+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:36.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:35.628+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:36.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:35.629+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:36.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:36.076+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:36.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:36.076+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:36.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:36.076+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:36.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:36.077+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:36.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:36.067+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:36.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:36.067+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:36.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:36.067+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:36.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:36.068+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:37.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:36.582+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:37.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:36.582+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:37.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:36.582+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:37.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:36.583+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:37.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:37.084+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:37.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:37.084+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:37.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:37.084+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:37.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:37.085+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:37.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:37.079+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:37.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:37.079+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:37.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:37.079+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:37.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:37.081+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:38.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:37.573+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:38.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:37.573+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:38.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:37.573+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:38.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:37.574+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:38.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:38.112+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:38.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:38.112+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:38.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:38.112+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:38.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:38.113+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:38.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:38.119+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:38.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:38.119+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:38.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:38.119+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:38.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:38.121+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:39.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:38.586+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:39.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:38.586+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:39.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:38.586+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:39.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:38.587+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:39.386 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:39.103+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:39.387 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:39.103+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:39.387 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:39.103+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:39.387 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:39.105+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:39.387 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:39.080+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:39.387 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:39.080+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:39.388 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:39.080+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:39.388 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:39.081+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:39.705 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:22:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:22:39.579+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 17421 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:22:39.705 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:39.584+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:39.705 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:39.584+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:39.705 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:39.584+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:39.706 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:39.585+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:40.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:40.105+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:40.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:40.105+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:40.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:40.105+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:40.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:40.107+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:40.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:40.061+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:40.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:40.061+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:40.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:40.061+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:40.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:40.062+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:41.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:40.589+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:41.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:40.589+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:41.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:40.589+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:41.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:40.590+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:41.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:41.081+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:41.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:41.081+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:41.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:41.081+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:41.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:41.082+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:41.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:41.046+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:41.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:41.046+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:41.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:41.046+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:41.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:41.047+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:42.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:41.562+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:42.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:41.562+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:42.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:41.562+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:42.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:41.563+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:42.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:42.073+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:42.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:42.073+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:42.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:42.073+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:42.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:42.075+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:42.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:42.052+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:42.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:42.052+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:42.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:42.052+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:42.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:42.053+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:43.006 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:42.568+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:43.006 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:42.568+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:43.006 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:42.568+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:43.006 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:42.569+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:43.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:43.056+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:43.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:43.056+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:43.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:43.056+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:43.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:43.057+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:43.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:43.036+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:43.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:43.036+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:43.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:43.036+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:43.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:43.037+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:43.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:43.586+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:43.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:43.586+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:43.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:43.586+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:43.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:43.587+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:44.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:44.050+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:44.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:44.050+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:44.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:44.050+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:44.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:44.051+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:44.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:44.004+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:44.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:44.004+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:44.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:44.004+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:44.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:44.005+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:45.028 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:22:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:22:44.581+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 17501 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:22:45.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:44.594+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:45.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:44.594+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:45.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:44.594+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:45.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:44.596+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:45.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:45.054+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:45.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:45.054+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:45.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:45.054+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:45.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:45.055+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:45.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:45.037+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:45.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:45.037+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:45.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:45.037+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:45.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:45.038+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:46.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:45.576+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:46.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:45.576+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:46.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:45.576+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:46.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:45.576+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:46.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:46.035+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:46.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:46.035+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:46.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:46.035+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:46.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:46.036+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:46.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:46.084+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:46.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:46.084+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:46.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:46.084+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:46.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:46.085+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:47.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:46.612+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:47.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:46.612+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:47.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:46.612+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:47.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:46.613+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:47.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:47.032+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:47.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:47.032+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:47.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:47.032+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:47.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:47.033+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:47.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:47.047+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:47.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:47.047+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:47.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:47.047+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:47.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:47.047+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:48.015 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:47.581+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:48.015 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:47.581+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:48.015 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:47.581+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:48.015 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:47.582+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:48.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:48.026+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:48.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:48.026+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:48.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:48.026+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:48.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:48.027+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:48.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:48.012+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:48.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:48.012+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:48.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:48.012+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:48.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:48.013+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:48.998 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:48.579+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:48.998 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:48.579+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:48.998 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:48.579+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:48.998 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:48.580+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:49.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:49.072+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:49.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:49.072+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:49.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:49.072+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:49.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:49.073+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:49.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:48.995+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:49.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:48.995+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:49.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:48.995+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:49.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:48.996+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:49.709 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:49.581+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:49.709 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:49.581+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:49.709 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:49.581+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:49.709 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:49.583+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:49.709 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:22:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:22:49.583+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 17580 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:22:50.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:50.035+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:50.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:50.035+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:50.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:50.035+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:50.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:50.035+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:50.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:50.027+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:50.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:50.027+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:50.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:50.027+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:50.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:50.028+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:50.996 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:50.557+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:50.996 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:50.557+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:50.996 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:50.557+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:50.996 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:50.557+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:51.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:50.993+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:51.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:50.993+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:51.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:50.993+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:51.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:50.994+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:51.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:51.066+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:51.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:51.066+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:51.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:51.066+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:51.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:51.067+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:52.012 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:51.551+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:52.012 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:51.551+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:52.013 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:51.551+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:52.013 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:51.553+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:52.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:52.010+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:52.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:52.010+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:52.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:52.010+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:52.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:52.010+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:52.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:52.092+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:52.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:52.092+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:52.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:52.092+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:52.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:52.093+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:53.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:52.543+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:53.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:52.543+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:53.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:52.543+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:53.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:52.544+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:53.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:53.030+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:53.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:53.030+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:53.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:53.030+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:53.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:53.031+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:53.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:53.083+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:53.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:53.083+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:53.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:53.083+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:53.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:53.083+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:54.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:53.588+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:54.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:53.588+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:54.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:53.588+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:54.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:53.588+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:54.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:54.059+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:54.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:54.059+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:54.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:54.059+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:54.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:54.060+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:54.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:54.054+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:54.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:54.054+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:54.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:54.054+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:54.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:54.055+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:55.028 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:22:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:22:54.585+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 17660 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:22:55.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:54.547+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:55.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:54.547+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:55.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:54.547+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:55.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:54.548+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:55.510 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:55.099+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:55.510 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:55.099+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:55.510 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:55.099+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:55.510 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:55.100+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:55.511 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:55.067+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:55.511 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:55.067+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:55.511 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:55.067+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:55.511 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:55.068+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:55.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:55.507+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:55.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:55.507+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:55.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:55.507+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:55.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:55.508+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:56.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:56.048+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:56.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:56.048+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:56.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:56.048+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:56.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:56.049+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:56.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:56.056+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:56.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:56.056+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:56.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:56.056+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:56.531 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:56.057+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:57.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:56.527+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:57.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:56.527+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:57.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:56.527+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:57.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:56.528+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:57.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:57.095+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:57.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:57.095+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:57.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:57.095+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:57.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:57.095+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:57.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:57.101+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:57.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:57.101+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:57.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:57.101+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:57.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:57.101+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:58.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:57.574+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:58.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:57.575+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:58.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:57.575+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:58.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:57.576+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:58.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:58.047+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:58.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:58.047+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:58.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:58.047+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:58.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:58.048+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:58.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:58.115+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:58.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:58.115+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:58.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:58.115+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:58.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:58.115+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:59.006 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:58.578+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:59.006 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:58.578+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:59.007 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:58.578+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:59.007 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:58.579+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:59.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:59.070+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:59.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:59.070+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:59.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:59.070+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:22:59.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:22:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:22:59.071+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:59.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:59.004+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:59.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:59.004+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:59.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:59.004+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:22:59.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:59.005+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:22:59.713 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:22:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:22:59.587+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 17740 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:22:59.713 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:59.602+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:59.713 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:59.602+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:59.713 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:59.602+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:22:59.714 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:22:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:22:59.602+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:00.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:59.962+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:00.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:59.962+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:00.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:59.962+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:00.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:22:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:22:59.963+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:00.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:00.031+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:00.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:00.031+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:00.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:00.031+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:00.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:00.032+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:01.010 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:00.568+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:01.011 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:00.568+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:01.011 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:00.568+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:01.011 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:00.568+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:01.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:01.008+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:01.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:01.008+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:01.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:01.008+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:01.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:01.009+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:01.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:01.058+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:01.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:01.058+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:01.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:01.058+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:01.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:01.059+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:01.925 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:01.552+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:01.925 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:01.552+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:01.925 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:01.552+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:01.925 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:01.553+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:02.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:02.047+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:02.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:02.047+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:02.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:02.047+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:02.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:02.048+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:02.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:01.987+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:02.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:01.987+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:02.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:01.987+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:02.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:01.988+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:03.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:02.593+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:03.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:02.593+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:03.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:02.593+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:03.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:02.594+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:03.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:03.033+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:03.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:03.033+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:03.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:03.033+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:03.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:03.034+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:03.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:03.043+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:03.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:03.043+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:03.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:03.043+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:03.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:03.044+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:04.020 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:03.583+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:04.020 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:03.583+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:04.020 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:03.583+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:04.021 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:03.583+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:04.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:04.060+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:04.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:04.060+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:04.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:04.060+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:04.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:04.060+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:04.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:04.018+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:04.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:04.018+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:04.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:04.018+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:04.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:04.019+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:04.986 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:23:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:23:04.589+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 17821 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:23:04.986 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:04.560+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:04.986 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:04.560+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:04.987 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:04.560+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:04.987 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:04.561+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:05.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:05.070+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:05.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:05.070+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:05.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:05.070+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:05.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:05.071+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:05.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:04.984+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:05.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:04.984+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:05.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:04.984+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:05.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:04.984+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:05.955 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:05.564+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:05.955 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:05.564+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:05.956 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:05.564+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:05.956 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:05.566+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:06.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:06.057+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:06.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:06.057+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:06.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:06.057+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:06.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:06.058+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:06.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:05.953+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:06.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:05.953+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:06.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:05.953+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:06.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:05.954+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:06.963 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:06.582+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:06.963 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:06.582+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:06.963 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:06.582+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:06.963 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:06.582+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:07.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:06.960+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:07.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:06.960+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:07.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:06.960+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:07.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:06.961+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:07.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:07.065+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:07.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:07.065+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:07.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:07.065+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:07.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:07.066+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:07.982 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:07.615+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:07.982 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:07.615+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:07.982 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:07.615+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:07.983 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:07.615+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:08.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:08.038+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:08.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:08.038+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:08.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:08.038+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:08.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:08.039+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:08.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:07.980+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:08.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:07.980+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:08.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:07.980+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:08.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:07.981+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:09.001 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:08.576+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:09.001 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:08.576+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:09.001 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:08.576+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:09.001 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:08.577+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:09.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:08.999+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:09.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:08.999+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:09.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:08.999+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:09.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:08.999+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:09.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:09.055+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:09.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:09.055+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:09.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:09.055+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:09.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:09.056+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:09.717 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:23:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:23:09.591+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 17902 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:23:09.717 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:09.598+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:09.718 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:09.598+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:09.718 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:09.598+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:09.718 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:09.599+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:10.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:09.961+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:10.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:09.961+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:10.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:09.961+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:10.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:09.961+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:10.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:10.103+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:10.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:10.103+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:10.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:10.103+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:10.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:10.104+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:11.004 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:10.563+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:11.004 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:10.563+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:11.004 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:10.563+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:11.005 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:10.563+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:11.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:11.139+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:11.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:11.139+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:11.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:11.139+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:11.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:11.140+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:11.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:11.002+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:11.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:11.002+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:11.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:11.002+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:11.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:11.003+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:11.960 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:11.607+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:11.961 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:11.607+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:11.961 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:11.607+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:11.961 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:11.608+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:12.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:12.125+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:12.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:12.125+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:12.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:12.125+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:12.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:12.126+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:12.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:11.958+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:12.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:11.958+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:12.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:11.958+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:12.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:11.959+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:13.008 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:12.632+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:13.008 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:12.632+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:13.008 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:12.632+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:13.008 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:12.634+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:13.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:13.157+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:13.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:13.157+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:13.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:13.157+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:13.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:13.158+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:13.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:13.005+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:13.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:13.005+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:13.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:13.005+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:13.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:13.006+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:14.024 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:13.677+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:14.024 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:13.677+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:14.024 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:13.677+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:14.024 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:13.678+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:14.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:14.022+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:14.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:14.022+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:14.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:14.022+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:14.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:14.023+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:14.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:14.139+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:14.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:14.139+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:14.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:14.139+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:14.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:14.140+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:14.979 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:23:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:23:14.593+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 17983 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:23:14.979 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:14.677+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:14.979 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:14.677+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:14.979 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:14.677+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:14.980 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:14.678+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:15.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:15.167+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:15.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:15.167+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:15.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:15.167+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:15.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:15.168+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:15.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:14.980+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:15.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:14.980+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:15.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:14.980+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:15.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:14.981+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:16.004 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:15.659+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:16.004 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:15.659+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:16.005 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:15.659+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:16.005 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:15.660+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:16.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:16.203+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:16.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:16.203+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:16.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:16.203+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:16.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:16.204+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:16.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:16.001+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:16.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:16.001+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:16.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:16.001+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:16.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:16.003+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:17.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:16.689+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:17.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:16.689+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:17.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:16.689+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:17.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:16.690+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:17.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:17.229+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:17.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:17.229+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:17.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:17.229+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:17.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:17.229+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:17.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:17.031+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:17.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:17.031+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:17.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:17.031+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:17.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:17.032+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:18.015 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:17.698+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:18.015 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:17.698+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:18.015 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:17.698+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:18.015 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:17.698+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:18.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:18.228+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:18.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:18.228+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:18.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:18.228+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:18.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:18.229+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:18.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:18.012+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:18.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:18.012+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:18.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:18.012+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:18.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:18.013+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:19.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:18.708+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:19.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:18.708+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:19.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:18.708+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:19.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:18.709+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:19.386 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:19.061+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:19.386 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:19.061+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:19.387 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:19.061+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:19.387 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:19.062+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:19.387 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:19.269+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:19.387 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:19.269+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:19.387 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:19.269+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:19.387 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:19.270+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:19.667 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:23:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:23:19.595+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 18064 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:23:20.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:19.665+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:20.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:19.665+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:20.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:19.665+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:20.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:19.666+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:20.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:20.253+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:20.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:20.253+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:20.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:20.253+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:20.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:20.254+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:20.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:20.096+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:20.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:20.096+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:20.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:20.096+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:20.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:20.097+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:21.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:20.676+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:21.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:20.676+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:21.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:20.676+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:21.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:20.677+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:21.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:21.053+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:21.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:21.053+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:21.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:21.053+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:47.352622+0000 (oldest deadline 2024-03-11T23:13:07.352622+0000) 2024-03-11T23:23:21.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:21.054+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:21.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:21.267+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:21.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:21.267+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:21.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:21.267+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:21.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:21.268+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:22.021 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:21.634+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:22.022 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:21.634+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:22.022 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:21.634+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:22.022 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:21.635+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:22.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:22.019+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:22.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:22.245+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:22.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:22.245+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:22.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:22.245+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:22.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:22.246+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:23.008 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:22.589+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:23.008 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:22.589+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:23.009 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:22.589+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:23.009 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:22.590+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:23.258 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:23.006+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:23.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:23.256+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:23.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:23.256+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:23.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:23.256+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:23.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:23.257+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:23.985 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:23.577+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:23.985 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:23.577+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:23.986 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:23.577+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:23.986 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:23.578+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:24.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:23.983+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:24.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:24.229+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:24.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:24.229+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:24.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:24.229+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:24.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:24.230+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:25.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:23:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:23:24.597+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 18144 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:23:25.002 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:24.565+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:25.002 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:24.565+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:25.002 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:24.565+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:25.003 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:24.565+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:25.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:24.999+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:25.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:25.241+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:25.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:25.241+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:25.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:25.241+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:25.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:25.242+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:26.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:25.532+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:26.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:25.532+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:26.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:25.532+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:26.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:25.532+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:26.500 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:26.029+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:26.500 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:26.218+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:26.500 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:26.218+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:26.501 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:26.218+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:26.501 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:26.218+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:26.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:26.508+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:26.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:26.508+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:26.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:26.508+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:26.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:26.509+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:27.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:27.029+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:27.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:27.253+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:27.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:27.253+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:27.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:27.253+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:27.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:27.254+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:28.013 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:27.548+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:28.013 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:27.548+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:28.013 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:27.548+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:28.013 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:27.549+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:28.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:28.011+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:28.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:28.223+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:28.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:28.223+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:28.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:28.223+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:28.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:28.224+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:28.968 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:28.544+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:28.969 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:28.544+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:28.969 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:28.544+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:28.969 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:28.545+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:29.273 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:28.966+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:29.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:29.271+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:29.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:29.271+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:29.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:29.271+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:29.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:29.271+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:29.933 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:23:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:23:29.598+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 18224 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:23:29.934 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:29.539+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:29.934 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:29.539+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:29.934 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:29.539+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:29.934 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:29.539+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:30.265 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:29.931+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:30.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:30.263+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:30.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:30.263+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:30.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:30.263+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:30.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:30.263+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:30.942 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:30.562+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:30.943 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:30.562+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:30.943 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:30.562+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:30.943 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:30.563+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:31.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:30.940+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:31.596 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:31.310+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:31.596 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:31.310+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:31.596 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:31.310+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:31.596 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:31.311+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:31.942 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:31.594+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:31.942 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:31.594+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:31.942 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:31.594+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:31.943 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:31.594+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:32.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:31.940+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:32.627 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:32.324+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:32.628 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:32.324+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:32.628 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:32.324+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:12:55.370030+0000 (oldest deadline 2024-03-11T23:13:15.370030+0000) 2024-03-11T23:23:32.628 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:32.325+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:32.944 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:32.625+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:32.944 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:32.625+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:32.944 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:32.625+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:32.944 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:32.626+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:33.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:32.942+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:33.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:33.367+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:33.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:33.578+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:33.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:33.578+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:33.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:33.578+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:33.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:33.579+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:34.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:33.901+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:34.603 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:34.557+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:34.603 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:34.557+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:34.603 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:34.557+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:34.603 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:34.558+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:34.604 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:34.333+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:34.856 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:23:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:23:34.600+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 18304 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:23:35.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:34.854+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:35.716 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:35.343+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:35.716 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:35.550+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:35.717 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:35.550+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:35.717 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:35.550+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:35.717 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:35.551+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:36.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:35.862+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:36.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:36.353+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:36.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:36.535+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:36.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:36.535+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:36.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:36.535+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:13:00.763452+0000 (oldest deadline 2024-03-11T23:13:20.763452+0000) 2024-03-11T23:23:36.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:36.536+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:37.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:36.907+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:37.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:37.383+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:37.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:37.541+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:38.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:37.942+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:38.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:38.383+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:38.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:38.556+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:39.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:38.940+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:39.605 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:39.346+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:39.967 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:23:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:23:39.602+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 18385 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:23:39.968 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:39.606+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:40.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:39.965+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:40.630 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:40.303+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:40.923 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:40.628+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:41.258 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:40.921+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:41.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:41.256+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:41.912 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:41.631+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:42.221 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:41.910+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:42.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:42.219+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:42.896 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:42.634+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:43.209 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:42.894+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:43.209 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:42.894+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:43.209 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:42.894+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:43.209 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:42.895+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:43.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:43.207+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:43.879 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:43.638+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:44.225 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:43.877+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:44.226 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:43.877+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:44.226 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:43.877+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:44.226 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:43.878+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:44.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:44.223+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:44.902 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:23:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:23:44.604+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 18466 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:23:44.903 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:44.616+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:45.214 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:44.900+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:45.214 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:44.900+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:45.214 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:44.900+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:45.215 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:44.901+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:45.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:45.212+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:45.934 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:45.581+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:46.256 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:45.932+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:46.256 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:45.932+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:46.256 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:45.932+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:46.256 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:45.933+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:46.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:46.254+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:46.908 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:46.589+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:47.252 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:46.906+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:47.253 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:46.906+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:47.253 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:46.906+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:47.253 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:46.907+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:47.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:47.251+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:47.935 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:47.548+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:48.241 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:47.933+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:48.241 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:47.933+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:48.241 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:47.933+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:48.242 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:47.934+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:48.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:48.239+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:48.968 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:48.546+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:49.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:48.966+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:49.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:48.966+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:49.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:48.966+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:49.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:48.967+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:49.533 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:49.280+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:49.934 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:23:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:23:49.606+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 18547 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:23:49.934 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:49.531+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:50.241 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:49.932+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:50.242 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:49.932+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:50.242 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:49.932+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:50.242 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:49.933+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:50.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:50.239+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:50.968 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:50.546+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:51.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:51.198+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:51.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:50.966+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:51.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:50.966+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:51.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:50.966+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:51.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:50.967+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:51.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:51.498+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:52.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:52.186+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:52.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:51.991+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:52.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:51.991+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:52.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:51.991+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:52.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:51.992+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:52.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:52.516+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:53.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:53.188+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:53.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:52.951+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:53.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:52.951+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:53.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:52.951+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:53.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:52.952+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:53.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:53.480+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:54.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:54.190+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:54.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:53.949+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:54.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:53.949+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:54.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:53.949+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:54.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:53.950+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:54.778 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:23:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:23:54.608+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 18628 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:23:54.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:54.504+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:55.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:55.216+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:55.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:54.994+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:55.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:54.994+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:55.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:54.994+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:55.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:54.995+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:56.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:55.527+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:56.522 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:56.246+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:23:56.523 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:56.246+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:23:56.523 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:56.246+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:23:56.523 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:56.247+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:56.523 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:56.028+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:56.523 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:56.028+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:56.524 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:56.028+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:56.524 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:56.029+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:56.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:56.520+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:57.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:57.253+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:23:57.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:57.253+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:23:57.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:57.253+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:23:57.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:57.254+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:57.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:57.020+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:57.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:57.020+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:57.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:57.020+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:57.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:57.021+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:57.977 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:57.555+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:58.256 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:57.975+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:58.257 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:57.975+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:58.257 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:57.975+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:58.257 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:57.976+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:58.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:58.254+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:23:58.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:58.254+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:23:58.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:58.254+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:23:58.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:58.255+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:59.017 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:58.579+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:23:59.017 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:58.579+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:23:59.017 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:58.579+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:23:59.017 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:58.580+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:59.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:59.218+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:23:59.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:59.218+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:23:59.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:59.218+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:23:59.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:23:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:23:59.219+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:59.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:59.015+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:59.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:59.015+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:59.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:59.015+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:23:59.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:23:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:23:59.016+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:23:59.736 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:23:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:23:59.610+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 18709 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:23:59.736 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:59.628+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:23:59.736 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:59.628+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:23:59.736 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:59.628+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:23:59.737 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:23:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:23:59.629+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:00.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:00.034+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:00.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:00.034+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:00.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:00.034+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:00.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:00.035+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:00.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:00.239+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:00.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:00.239+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:00.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:00.239+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:00.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:00.239+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:01.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:00.666+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:01.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:00.666+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:01.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:00.666+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:01.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:00.667+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:01.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:01.025+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:01.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:01.025+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:01.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:01.025+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:01.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:01.026+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:01.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:01.208+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:01.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:01.208+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:01.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:01.208+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:01.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:01.209+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:02.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:01.717+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:02.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:01.717+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:02.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:01.717+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:02.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:01.717+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:02.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:02.206+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:02.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:02.206+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:02.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:02.206+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:02.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:02.206+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:02.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:02.027+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:02.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:02.027+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:02.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:02.027+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:02.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:02.027+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:03.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:02.731+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:03.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:02.731+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:03.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:02.731+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:03.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:02.732+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:03.365 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:03.065+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:03.365 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:03.065+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:03.365 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:03.065+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:03.366 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:03.065+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:03.366 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:03.173+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:03.366 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:03.174+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:03.366 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:03.174+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:03.366 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:03.174+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:04.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:03.689+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:04.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:03.689+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:04.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:03.689+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:04.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:03.690+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:04.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:04.194+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:04.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:04.194+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:04.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:04.194+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:04.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:04.194+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:04.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:04.109+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:04.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:04.109+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:04.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:04.109+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:04.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:04.110+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:05.028 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:24:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:24:04.612+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 18790 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:24:05.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:04.739+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:05.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:04.739+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:05.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:04.739+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:05.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:04.739+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:05.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:05.205+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:05.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:05.205+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:05.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:05.205+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:05.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:05.205+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:05.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:05.140+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:05.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:05.140+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:05.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:05.140+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:05.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:05.141+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:06.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:05.772+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:06.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:05.772+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:06.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:05.772+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:06.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:05.773+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:06.436 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:06.184+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:06.436 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:06.184+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:06.437 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:06.184+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:06.437 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:06.185+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:06.437 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:06.180+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:06.437 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:06.180+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:06.437 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:06.180+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:06.438 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:06.181+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:07.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:06.749+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:07.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:06.749+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:07.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:06.749+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:07.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:06.750+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:07.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:07.211+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:07.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:07.211+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:07.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:07.211+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:07.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:07.212+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:07.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:07.168+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:07.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:07.168+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:07.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:07.168+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:07.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:07.168+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:08.123 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:07.796+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:08.124 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:07.796+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:08.124 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:07.796+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:08.124 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:07.797+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:08.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:08.170+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:08.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:08.170+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:08.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:08.170+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:08.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:08.171+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:08.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:08.121+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:08.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:08.121+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:08.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:08.121+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:08.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:08.122+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:09.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:08.770+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:09.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:08.770+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:09.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:08.770+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:09.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:08.770+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:09.386 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:09.084+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:09.386 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:09.084+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:09.386 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:09.084+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:09.386 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:09.084+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:09.386 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:09.133+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:09.386 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:09.133+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:09.387 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:09.133+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:09.387 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:09.133+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:09.641 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:24:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:24:09.614+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 18872 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:24:10.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:09.764+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:10.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:09.764+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:10.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:09.764+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:10.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:09.765+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:10.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:10.123+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:10.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:10.123+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:10.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:10.123+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:10.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:10.124+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:10.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:10.106+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:10.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:10.106+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:10.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:10.106+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:10.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:10.107+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:11.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:10.766+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:11.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:10.766+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:11.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:10.766+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:11.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:10.767+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:11.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:11.143+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:11.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:11.143+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:11.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:11.143+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:11.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:11.144+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:11.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:11.133+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:11.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:11.133+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:11.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:11.133+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:11.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:11.134+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:12.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:11.726+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:12.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:11.726+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:12.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:11.726+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:12.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:11.727+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:12.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:12.160+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:12.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:12.160+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:12.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:12.160+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:12.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:12.161+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:12.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:12.159+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:12.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:12.159+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:12.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:12.159+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:12.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:12.160+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:13.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:12.729+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:13.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:12.729+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:13.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:12.729+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:13.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:12.731+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:13.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:13.151+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:13.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:13.151+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:13.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:13.151+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:13.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:13.151+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:13.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:13.121+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:13.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:13.121+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:13.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:13.121+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:13.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:13.122+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:14.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:13.758+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:14.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:13.758+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:14.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:13.758+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:14.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:13.759+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:14.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:14.172+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:14.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:14.172+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:14.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:14.172+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:14.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:14.172+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:14.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:14.151+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:14.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:14.151+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:14.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:14.151+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:14.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:14.151+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:15.028 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:24:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:24:14.616+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 18955 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:24:15.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:14.808+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:15.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:14.808+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:15.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:14.808+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:15.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:14.809+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:15.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:15.174+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:15.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:15.174+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:15.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:15.174+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:15.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:15.175+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:15.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:15.103+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:15.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:15.103+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:15.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:15.103+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:15.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:15.104+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:16.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:15.825+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:16.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:15.825+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:16.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:15.825+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:16.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:15.825+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:16.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:16.139+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:16.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:16.139+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:16.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:16.139+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:16.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:16.139+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:16.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:16.065+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:16.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:16.065+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:16.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:16.065+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:16.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:16.066+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:17.091 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:16.814+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:17.091 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:16.814+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:17.092 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:16.814+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:17.092 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:16.814+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:17.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:17.137+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:17.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:17.137+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:17.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:17.137+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:17.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:17.138+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:17.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:17.089+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:17.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:17.089+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:17.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:17.089+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:17.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:17.090+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:18.115 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:17.838+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:18.115 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:17.838+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:18.115 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:17.838+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:18.116 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:17.839+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:18.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:18.122+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:18.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:18.122+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:18.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:18.122+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:18.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:18.123+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:18.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:18.113+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:18.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:18.113+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:18.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:18.113+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:18.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:18.114+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:19.105 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:18.796+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:19.106 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:18.796+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:19.106 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:18.796+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:19.106 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:18.796+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:19.387 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:19.161+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:19.387 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:19.161+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:19.387 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:19.161+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:19.387 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:19.162+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:19.388 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:19.103+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:19.388 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:19.103+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:19.388 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:19.103+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:19.388 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:19.104+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:19.642 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:24:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:24:19.618+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 19038 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:24:20.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:19.759+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:20.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:19.759+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:20.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:19.759+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:20.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:19.760+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:20.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:20.074+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:20.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:20.074+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:20.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:20.074+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:20.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:20.075+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:20.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:20.178+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:20.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:20.178+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:20.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:20.178+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:20.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:20.179+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:21.092 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:20.801+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:21.092 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:20.801+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:21.092 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:20.801+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:21.092 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:20.802+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:21.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:21.090+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:21.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:21.090+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:21.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:21.090+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:21.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:21.090+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:21.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:21.219+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:21.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:21.219+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:21.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:21.219+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:21.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:21.220+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:22.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:21.772+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:22.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:21.772+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:22.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:21.772+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:22.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:21.773+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:22.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:22.044+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:22.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:22.044+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:22.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:22.044+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:22.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:22.045+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:22.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:22.199+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:22.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:22.199+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:22.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:22.199+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:22.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:22.200+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:23.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:22.772+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:23.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:22.772+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:23.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:22.772+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:23.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:22.772+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:23.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:23.070+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:23.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:23.070+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:23.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:23.070+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:23.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:23.071+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:23.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:23.198+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:23.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:23.198+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:23.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:23.198+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:23.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:23.199+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:24.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:23.812+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:24.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:23.812+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:24.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:23.812+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:24.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:23.813+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:24.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:24.050+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:24.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:24.050+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:24.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:24.050+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:24.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:24.050+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:24.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:24.238+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:24.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:24.238+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:24.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:24.238+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:24.529 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:24.239+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:25.016 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:24:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:24:24.620+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 19121 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:24:25.016 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:24.776+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:25.017 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:24.776+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:25.017 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:24.776+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:25.017 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:24.776+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:25.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:25.014+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:25.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:25.014+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:25.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:25.014+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:25.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:25.015+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:25.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:25.251+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:25.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:25.251+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:25.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:25.251+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:25.280 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:25.252+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:26.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:25.987+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:26.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:25.987+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:26.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:25.987+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:26.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:25.988+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:26.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:25.755+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:26.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:25.755+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:26.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:25.755+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:26.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:25.756+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:26.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:26.246+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:26.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:26.246+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:26.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:26.246+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:26.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:26.246+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:27.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:26.950+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:27.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:26.950+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:27.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:26.950+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:27.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:26.950+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:27.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:26.753+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:27.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:26.753+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:27.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:26.753+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:27.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:26.754+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:27.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:27.230+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:27.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:27.230+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:27.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:27.230+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:27.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:27.231+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:28.270 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:27.790+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:28.271 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:27.790+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:28.271 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:27.790+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:28.271 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:27.791+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:28.271 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:27.950+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:28.271 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:27.950+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:28.271 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:27.950+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:28.272 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:27.950+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:28.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:28.269+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:28.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:28.269+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:28.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:28.269+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:28.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:28.269+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:29.231 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:28.941+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:29.232 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:28.941+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:29.232 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:28.941+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:29.232 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:28.942+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:29.232 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:28.818+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:29.233 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:28.818+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:29.233 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:28.819+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:29.233 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:28.819+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:29.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:29.229+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:29.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:29.229+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:29.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:29.229+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:29.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:29.230+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:29.925 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:24:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:24:29.622+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 19204 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:24:29.926 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:29.791+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:29.926 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:29.791+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:29.926 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:29.791+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:29.926 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:29.792+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:30.260 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:29.923+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:30.261 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:29.923+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:30.261 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:29.923+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:30.261 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:29.924+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:30.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:30.258+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:30.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:30.258+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:30.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:30.258+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:30.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:30.259+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:31.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:30.810+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:31.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:30.810+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:31.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:30.810+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:31.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:30.811+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:31.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:30.923+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:31.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:30.923+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:31.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:30.923+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:31.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:30.924+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:31.714 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:31.294+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:31.714 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:31.294+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:31.714 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:31.294+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:31.714 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:31.295+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:32.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:31.840+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:32.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:31.840+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:32.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:31.840+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:32.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:31.841+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:32.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:31.951+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:32.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:31.951+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:32.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:31.951+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:32.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:31.952+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:32.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:32.312+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:32.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:32.312+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:32.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:32.312+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:32.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:32.313+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:33.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:32.885+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:33.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:32.885+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:33.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:32.885+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:33.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:32.885+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:33.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:32.973+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:33.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:32.973+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:33.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:32.973+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:33.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:32.974+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:33.596 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:33.302+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:33.597 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:33.302+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:33.597 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:33.302+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:33.597 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:33.303+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:34.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:33.900+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:34.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:33.900+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:34.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:33.900+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:34.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:33.901+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:34.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:33.978+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:34.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:33.978+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:34.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:33.978+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:34.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:33.978+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:34.627 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:34.325+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:34.628 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:34.325+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:34.628 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:34.325+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:34.628 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:34.326+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:34.920 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:24:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:24:34.624+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 19286 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:24:35.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:34.919+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:35.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:34.919+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:35.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:34.919+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:35.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:34.920+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:35.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:34.936+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:35.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:34.936+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:35.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:34.936+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:35.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:34.936+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:35.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:35.346+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:35.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:35.346+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:35.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:35.346+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:35.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:35.347+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:36.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:35.947+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:36.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:35.947+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:36.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:35.947+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:36.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:35.948+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:36.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:35.936+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:36.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:35.936+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:36.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:35.936+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:36.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:35.936+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:36.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:36.324+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:36.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:36.324+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:36.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:36.324+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:36.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:36.325+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:37.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:36.951+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:37.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:36.951+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:37.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:36.951+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:37.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:36.951+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:37.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:36.908+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:37.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:36.908+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:37.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:36.908+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:37.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:36.909+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:37.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:37.289+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:37.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:37.289+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:37.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:37.289+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:37.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:37.290+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:38.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:37.925+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:38.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:37.925+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:38.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:37.925+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:38.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:37.925+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:38.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:37.900+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:38.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:37.900+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:38.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:37.900+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:38.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:37.902+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:38.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:38.282+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:38.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:38.282+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:38.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:38.282+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:38.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:38.283+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:39.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:38.944+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:39.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:38.944+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:39.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:38.944+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:39.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:38.945+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:39.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:38.883+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:39.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:38.883+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:39.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:38.883+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:39.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:38.883+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:39.628 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:39.332+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:39.629 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:39.332+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:39.629 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:39.332+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:39.629 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:39.332+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:39.947 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:24:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:24:39.626+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 19368 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:24:39.947 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:39.875+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:39.948 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:39.875+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:39.948 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:39.875+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:39.948 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:39.876+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:40.228 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:39.945+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:40.228 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:39.945+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:40.228 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:39.945+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:40.228 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:39.946+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:40.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:40.376+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:40.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:40.376+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:40.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:40.376+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:40.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:40.377+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:41.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:40.862+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:41.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:40.862+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:41.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:40.862+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:41.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:40.863+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:41.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:40.957+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:41.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:40.957+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:41.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:40.957+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:41.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:40.958+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:41.714 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:41.332+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:41.714 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:41.332+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:41.714 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:41.332+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:41.715 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:41.333+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:42.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:41.875+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:42.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:41.875+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:42.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:41.875+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:42.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:41.875+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:42.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:41.935+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:42.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:41.935+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:42.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:41.935+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:42.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:41.935+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:42.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:42.313+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:42.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:42.313+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:42.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:42.313+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:42.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:42.314+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:43.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:42.922+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:43.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:42.922+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:43.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:42.922+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:43.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:42.923+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:43.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:42.965+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:43.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:42.965+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:43.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:42.965+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:43.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:42.966+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:43.597 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:43.310+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:43.597 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:43.310+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:43.597 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:43.310+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:43.597 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:43.310+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:44.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:43.960+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:44.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:43.960+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:44.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:43.960+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:44.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:43.961+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:44.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:43.954+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:44.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:43.954+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:44.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:43.954+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:44.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:43.955+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:44.630 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:44.296+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:44.631 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:44.296+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:44.631 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:44.296+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:44.631 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:44.297+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:44.931 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:24:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:24:44.628+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 19452 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:24:45.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:44.931+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:45.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:44.931+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:45.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:44.931+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:45.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:44.932+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:45.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:44.930+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:45.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:44.930+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:45.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:44.930+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:45.281 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:44.930+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:45.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:45.319+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:45.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:45.319+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:45.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:45.319+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:45.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:45.320+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:46.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:45.885+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:46.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:45.885+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:46.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:45.885+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:46.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:45.886+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:46.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:45.950+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:46.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:45.950+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:46.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:45.950+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:46.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:45.951+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:46.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:46.328+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:46.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:46.328+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:46.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:46.328+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:46.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:46.329+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:47.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:46.839+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:47.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:46.839+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:47.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:46.839+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:47.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:46.840+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:47.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:46.947+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:47.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:46.947+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:47.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:46.947+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:47.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:46.947+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:47.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:47.357+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:47.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:47.357+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:47.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:47.357+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:47.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:47.357+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:48.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:47.792+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:48.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:47.792+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:48.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:47.792+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:48.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:47.793+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:48.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:47.972+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:48.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:47.972+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:48.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:47.972+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:48.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:47.973+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:48.750 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:48.385+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:48.751 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:48.385+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:48.751 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:48.385+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:48.751 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:48.386+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:49.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:48.951+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:49.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:48.951+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:49.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:48.951+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:49.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:48.951+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:49.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:48.748+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:49.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:48.748+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:49.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:48.748+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:49.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:48.749+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:49.645 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:24:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:24:49.631+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 19534 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:24:49.646 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:49.403+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:49.646 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:49.403+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:49.646 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:49.403+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:49.646 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:49.404+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:49.928 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:49.773+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:49.929 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:49.773+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:49.929 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:49.773+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:49.929 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:49.774+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:50.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:49.927+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:50.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:49.927+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:50.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:49.927+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:50.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:49.927+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:50.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:50.372+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:50.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:50.372+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:50.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:50.372+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:50.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:50.373+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:51.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:50.798+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:51.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:50.798+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:51.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:50.798+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:51.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:50.799+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:51.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:50.891+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:51.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:50.891+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:51.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:50.891+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:51.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:50.891+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:51.715 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:51.417+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:51.715 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:51.417+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:51.715 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:51.417+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:51.715 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:51.418+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:52.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:51.822+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:52.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:51.822+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:52.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:51.822+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:52.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:51.823+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:52.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:51.941+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:52.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:51.941+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:52.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:51.941+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:52.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:51.941+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:52.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:52.467+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:52.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:52.467+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:52.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:52.467+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:52.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:52.468+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:53.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:52.844+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:53.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:52.844+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:53.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:52.844+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:53.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:52.844+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:53.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:52.895+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:53.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:52.895+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:53.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:52.895+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:53.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:52.896+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:53.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:53.468+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:53.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:53.468+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:53.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:53.468+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:53.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:53.469+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:54.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:53.878+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:54.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:53.878+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:54.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:53.878+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:54.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:53.879+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:54.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:53.876+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:54.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:53.876+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:54.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:53.876+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:54.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:53.877+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:54.760 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:24:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:24:54.633+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 19618 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:24:54.760 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:54.427+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:54.760 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:54.427+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:54.760 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:54.427+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:54.760 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:54.428+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:55.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:54.922+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:55.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:54.922+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:55.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:54.922+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:55.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:54.923+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:55.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:54.873+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:55.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:54.873+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:55.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:54.873+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:55.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:54.874+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:55.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:55.464+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:55.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:55.464+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:55.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:55.464+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:55.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:55.465+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:56.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:55.913+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:56.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:55.913+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:56.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:55.913+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:56.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:55.913+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:56.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:55.849+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:56.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:55.849+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:56.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:55.849+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:56.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:55.850+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:56.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:56.488+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:56.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:56.488+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:56.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:56.488+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:56.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:56.489+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:57.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:56.879+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:57.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:56.879+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:57.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:56.879+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:57.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:56.880+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:57.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:56.815+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:57.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:56.815+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:57.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:56.815+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:57.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:56.815+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:57.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:57.522+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:57.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:57.522+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:57.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:57.522+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:57.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:57.522+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:58.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:57.823+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:58.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:57.823+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:58.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:57.823+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:58.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:57.823+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:58.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:57.928+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:58.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:57.928+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:58.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:57.928+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:58.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:57.929+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:58.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:58.508+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:58.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:58.508+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:58.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:58.508+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:58.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:58.508+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:59.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:58.973+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:59.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:58.973+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:59.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:58.973+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:24:59.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:58.974+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:59.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:58.866+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:59.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:58.866+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:59.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:58.866+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:59.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:58.867+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:59.637 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:59.517+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:59.637 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:59.517+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:59.637 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:59.517+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:24:59.638 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:24:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:24:59.518+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:24:59.988 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:24:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:24:59.635+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 19700 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:24:59.989 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:59.864+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:59.989 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:59.864+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:59.989 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:59.864+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:24:59.989 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:24:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:24:59.864+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:00.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:59.987+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:00.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:59.987+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:00.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:59.987+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:00.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:24:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:24:59.988+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:00.830 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:00.536+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:00.830 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:00.536+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:00.830 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:00.536+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:00.831 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:00.537+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:01.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:00.828+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:01.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:00.828+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:01.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:00.828+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:01.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:00.829+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:01.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:00.973+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:01.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:00.973+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:01.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:00.973+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:01.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:00.973+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:01.781 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:01.519+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:01.783 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:01.519+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:01.783 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:01.519+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:01.783 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:01.519+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:02.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:01.952+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:02.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:01.953+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:02.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:01.953+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:02.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:01.954+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:02.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:01.845+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:02.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:01.845+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:02.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:01.845+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:02.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:01.845+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:02.822 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:02.552+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:02.822 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:02.552+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:02.822 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:02.552+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:02.823 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:02.553+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:03.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:02.820+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:03.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:02.820+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:03.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:02.820+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:03.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:02.821+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:03.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:02.943+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:03.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:02.943+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:03.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:02.943+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:03.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:02.944+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:03.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:03.514+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:03.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:03.514+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:03.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:03.514+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:03.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:03.515+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:04.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:03.847+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:04.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:03.847+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:04.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:03.847+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:04.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:03.847+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:04.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:03.964+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:04.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:03.964+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:04.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:03.964+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:04.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:03.964+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:04.765 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:25:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:25:04.637+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 19784 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:25:04.766 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:04.494+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:04.766 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:04.494+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:04.766 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:04.494+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:04.766 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:04.495+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:05.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:04.998+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:05.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:04.998+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:05.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:04.998+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:05.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:04.999+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:05.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:04.891+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:05.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:04.891+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:05.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:04.891+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:05.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:04.891+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:05.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:05.460+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:05.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:05.460+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:05.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:05.460+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:05.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:05.461+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:06.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:05.978+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:06.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:05.978+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:06.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:05.978+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:06.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:05.978+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:06.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:05.901+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:06.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:05.901+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:06.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:05.901+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:06.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:05.902+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:06.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:06.499+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:06.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:06.499+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:06.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:06.499+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:06.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:06.501+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:07.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:06.938+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:07.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:06.938+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:07.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:06.938+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:07.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:06.939+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:07.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:07.006+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:07.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:07.006+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:07.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:07.006+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:07.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:07.007+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:07.897 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:07.528+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:07.897 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:07.528+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:07.898 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:07.528+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:07.898 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:07.529+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:08.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:08.037+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:08.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:08.037+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:08.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:08.037+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:08.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:08.038+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:08.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:07.895+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:08.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:07.895+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:08.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:07.895+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:08.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:07.896+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:08.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:08.489+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:08.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:08.489+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:08.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:08.489+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:08.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:08.489+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:09.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:08.923+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:09.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:08.923+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:09.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:08.923+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:09.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:08.925+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:09.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:09.071+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:09.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:09.071+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:09.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:09.071+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:09.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:09.072+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:09.641 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:09.497+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:09.641 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:09.497+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:09.641 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:09.497+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:09.642 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:09.498+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:09.909 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:25:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:25:09.639+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 19866 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:25:10.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:10.090+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:10.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:10.090+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:10.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:10.090+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:10.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:10.091+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:10.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:09.908+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:10.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:09.908+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:10.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:09.908+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:10.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:09.909+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:10.883 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:10.529+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:10.884 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:10.529+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:10.884 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:10.529+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:10.884 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:10.531+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:11.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:10.882+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:11.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:10.882+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:11.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:10.882+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:11.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:10.883+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:11.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:11.110+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:11.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:11.110+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:11.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:11.110+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:11.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:11.111+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:11.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:11.488+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:11.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:11.488+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:11.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:11.488+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:11.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:11.489+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:12.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:12.078+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:12.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:12.078+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:12.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:12.078+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:12.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:12.079+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:12.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:11.848+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:12.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:11.848+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:12.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:11.848+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:12.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:11.849+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:12.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:12.449+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:12.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:12.449+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:12.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:12.449+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:12.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:12.450+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:13.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:13.057+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:13.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:13.057+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:13.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:13.057+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:13.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:13.057+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:13.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:12.848+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:13.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:12.848+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:13.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:12.848+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:13.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:12.849+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:13.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:13.480+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:13.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:13.480+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:13.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:13.480+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:13.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:13.481+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:14.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:14.080+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:14.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:14.080+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:14.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:14.080+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:14.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:14.080+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:14.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:13.832+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:14.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:13.832+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:14.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:13.832+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:14.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:13.833+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:14.768 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:25:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:25:14.641+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 19950 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:25:14.768 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:14.490+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:14.768 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:14.490+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:14.768 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:14.490+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:14.769 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:14.491+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:15.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:14.815+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:15.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:14.815+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:15.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:14.815+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:15.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:14.816+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:15.497 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:15.069+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:15.497 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:15.069+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:15.497 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:15.069+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:15.497 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:15.071+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:15.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:15.495+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:15.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:15.495+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:15.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:15.495+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:15.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:15.496+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:16.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:16.044+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:16.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:16.044+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:16.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:16.044+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:16.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:16.044+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:16.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:15.823+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:16.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:15.823+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:16.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:15.823+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:16.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:15.824+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:16.809 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:16.536+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:16.809 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:16.536+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:16.809 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:16.536+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:16.809 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:16.537+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:17.075 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:16.807+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:17.075 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:16.807+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:17.075 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:16.807+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:17.075 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:16.808+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:17.504 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:17.073+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:17.504 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:17.073+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:17.504 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:17.073+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:17.505 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:17.074+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:17.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:17.502+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:17.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:17.502+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:17.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:17.502+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:17.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:17.503+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:18.113 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:17.813+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:18.113 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:17.813+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:18.113 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:17.813+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:18.113 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:17.814+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:18.479 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:18.111+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:18.479 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:18.111+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:18.479 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:18.111+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:18.479 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:18.112+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:18.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:18.477+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:18.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:18.477+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:18.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:18.477+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:18.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:18.478+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:19.086 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:18.831+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:19.086 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:18.831+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:19.086 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:18.831+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:19.086 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:18.832+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:19.385 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:19.084+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:19.385 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:19.084+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:19.385 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:19.084+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:19.386 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:19.085+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:19.646 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:19.501+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:19.646 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:19.501+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:19.646 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:19.501+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:19.646 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:19.502+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:20.027 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:25:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:25:19.644+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 20032 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:25:20.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:19.850+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:20.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:19.850+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:20.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:19.850+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:20.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:19.850+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:20.463 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:20.050+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:20.463 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:20.050+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:20.463 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:20.050+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:20.464 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:20.051+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:20.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:20.461+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:20.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:20.461+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:20.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:20.461+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:20.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:20.462+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:21.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:21.013+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:21.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:21.013+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:21.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:21.013+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:21.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:21.014+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:21.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:20.898+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:21.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:20.898+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:21.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:20.898+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:21.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:20.899+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:21.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:21.490+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:21.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:21.490+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:21.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:21.490+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:21.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:21.491+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:22.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:22.037+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:22.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:22.037+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:22.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:22.037+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:22.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:22.038+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:22.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:21.850+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:22.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:21.850+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:22.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:21.850+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:22.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:21.850+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:22.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:22.463+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:22.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:22.463+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:22.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:22.463+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:22.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:22.464+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:23.236 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:22.881+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:23.236 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:22.881+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:23.236 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:22.881+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:23.236 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:22.882+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:23.236 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:23.074+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:23.236 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:23.074+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:23.237 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:23.074+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:23.237 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:23.075+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:23.486 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:23.484+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:23.486 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:23.484+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:23.486 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:23.484+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:23.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:23.485+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:24.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:24.035+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:24.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:24.035+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:24.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:24.035+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:24.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:24.036+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:24.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:23.887+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:24.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:23.887+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:24.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:23.887+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:24.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:23.887+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:24.777 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:25:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:25:24.646+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 20113 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:25:24.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:24.462+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:24.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:24.462+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:24.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:24.462+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:24.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:24.463+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:25.080 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:24.849+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:25.080 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:24.849+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:25.081 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:24.849+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:25.081 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:24.850+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:25.475 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:25.078+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:25.475 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:25.078+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:25.475 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:25.078+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:25.475 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:25.079+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:25.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:25.473+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:25.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:25.473+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:25.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:25.473+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:25.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:25.474+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:26.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:26.104+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:26.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:26.104+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:26.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:26.104+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:26.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:26.106+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:26.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:25.859+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:26.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:25.859+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:26.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:25.859+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:26.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:25.860+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:26.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:26.470+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:26.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:26.470+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:26.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:26.470+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:26.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:26.470+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:27.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:27.125+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:27.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:27.125+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:27.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:27.125+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:27.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:27.126+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:27.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:26.896+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:27.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:26.896+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:27.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:26.896+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:27.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:26.897+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:27.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:27.494+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:27.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:27.494+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:27.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:27.494+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:27.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:27.496+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:28.118 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:27.849+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:28.118 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:27.849+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:28.119 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:27.849+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:28.119 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:27.850+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:28.468 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:28.116+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:28.469 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:28.117+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:28.469 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:28.117+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:28.469 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:28.118+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:28.469 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:28.467+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:28.469 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:28.467+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:28.469 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:28.467+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:28.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:28.468+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:29.148 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:28.857+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:29.148 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:28.857+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:29.148 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:28.857+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:29.148 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:28.858+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:29.448 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:29.146+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:29.448 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:29.146+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:29.448 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:29.146+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:29.449 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:29.147+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:29.776 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:25:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:25:29.648+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 20192 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:25:29.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:29.446+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:29.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:29.446+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:29.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:29.446+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:29.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:29.447+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:30.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:29.844+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:30.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:29.844+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:30.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:29.844+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:30.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:29.845+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:30.461 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:30.102+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:30.461 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:30.102+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:30.461 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:30.102+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:30.461 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:30.102+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:30.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:30.459+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:30.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:30.459+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:30.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:30.459+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:30.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:30.460+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:31.115 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:30.846+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:31.116 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:30.846+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:31.116 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:30.846+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:31.116 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:30.847+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:31.468 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:31.114+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:31.468 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:31.114+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:31.468 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:31.114+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:31.468 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:31.115+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:31.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:31.466+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:31.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:31.466+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:31.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:31.466+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:31.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:31.467+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:32.164 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:31.884+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:32.165 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:31.884+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:32.165 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:31.884+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:32.165 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:31.885+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:32.425 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:32.163+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:32.425 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:32.163+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:32.425 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:32.163+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:32.425 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:32.164+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:32.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:32.424+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:32.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:32.424+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:32.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:32.424+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:32.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:32.425+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:33.176 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:32.889+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:33.176 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:32.889+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:33.176 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:32.889+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:33.177 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:32.890+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:33.461 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:33.174+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:33.461 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:33.174+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:33.461 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:33.174+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:33.461 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:33.175+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:33.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:33.459+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:33.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:33.459+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:33.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:33.459+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:33.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:33.459+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:34.157 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:33.901+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:34.157 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:33.901+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:34.158 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:33.901+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:34.158 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:33.902+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:34.447 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:34.156+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:34.448 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:34.156+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:34.448 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:34.156+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:34.448 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:34.157+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:34.776 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:25:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:25:34.650+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 20270 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:25:34.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:34.446+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:34.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:34.446+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:34.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:34.446+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:34.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:34.446+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:35.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:34.891+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:35.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:34.891+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:35.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:34.891+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:35.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:34.892+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:35.456 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:35.169+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:35.456 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:35.169+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:35.456 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:35.169+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:35.457 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:35.170+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:35.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:35.454+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:35.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:35.454+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:35.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:35.454+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:35.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:35.455+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:36.218 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:35.937+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:36.218 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:35.937+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:36.218 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:35.937+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:36.218 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:35.938+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:36.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:36.410+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:36.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:36.410+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:36.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:36.410+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:36.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:36.411+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:36.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:36.216+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:36.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:36.216+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:36.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:36.216+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:36.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:36.217+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:37.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:37.183+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:37.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:37.183+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:37.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:37.183+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:37.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:37.184+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:37.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:36.945+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:37.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:36.945+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:37.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:36.945+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:37.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:36.946+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:37.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:37.364+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:37.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:37.364+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:37.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:37.364+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:37.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:37.365+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:38.187 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:37.919+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:38.187 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:37.919+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:38.188 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:37.919+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:38.188 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:37.920+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:38.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:38.186+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:38.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:38.186+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:38.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:38.186+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:38.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:38.187+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:38.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:38.397+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:38.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:38.397+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:38.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:38.397+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:38.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:38.398+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:39.222 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:38.897+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:39.222 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:38.897+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:39.222 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:38.897+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:39.222 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:38.898+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:39.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:39.412+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:39.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:39.412+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:39.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:39.412+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:39.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:39.413+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:39.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:39.220+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:39.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:39.220+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:39.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:39.220+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:39.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:39.221+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:40.027 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:25:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:25:39.652+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 20343 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:25:40.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:39.858+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:40.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:39.858+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:40.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:39.858+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:40.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:39.859+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:40.457 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:40.217+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:40.457 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:40.217+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:40.457 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:40.217+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:40.457 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:40.218+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:40.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:40.456+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:40.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:40.456+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:40.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:40.456+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:40.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:40.457+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:41.242 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:40.879+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:41.242 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:40.879+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:41.242 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:40.879+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:41.242 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:40.879+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:41.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:41.240+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:41.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:41.240+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:41.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:41.240+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:41.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:41.241+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:41.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:41.471+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:41.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:41.471+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:41.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:41.471+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:41.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:41.472+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:42.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:41.894+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:42.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:41.894+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:42.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:41.894+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:42.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:41.895+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:42.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:42.447+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:42.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:42.447+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:42.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:42.447+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:42.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:42.448+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:42.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:42.267+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:42.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:42.267+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:42.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:42.267+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:42.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:42.267+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:43.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:42.853+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:43.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:42.853+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:43.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:42.853+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:43.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:42.854+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:43.601 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:43.415+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:43.601 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:43.415+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:43.601 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:43.415+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:43.602 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:43.416+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:43.602 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:43.292+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:43.602 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:43.292+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:43.602 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:43.292+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:43.602 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:43.292+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:44.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:43.875+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:44.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:43.875+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:44.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:43.875+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:44.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:43.876+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:44.656 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:44.386+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:44.656 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:44.386+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:44.656 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:44.386+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:44.657 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:44.387+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:44.657 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:44.317+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:44.657 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:44.317+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:44.657 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:44.317+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:44.657 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:44.318+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:45.027 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:25:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:25:44.654+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 20421 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:25:45.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:44.900+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:45.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:44.900+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:45.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:44.900+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:45.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:44.901+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:45.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:45.361+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:45.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:45.361+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:45.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:45.361+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:45.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:45.362+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:45.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:45.376+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:45.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:45.376+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:45.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:45.376+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:45.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:45.377+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:46.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:45.909+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:46.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:45.909+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:46.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:45.909+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:46.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:45.910+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:46.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:46.380+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:46.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:46.380+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:46.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:46.380+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:46.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:46.381+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:46.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:46.412+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:46.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:46.412+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:46.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:46.412+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:46.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:46.413+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:47.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:46.958+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:47.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:46.958+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:47.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:46.958+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:47.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:46.959+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:47.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:47.371+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:47.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:47.371+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:47.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:47.371+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:47.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:47.371+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:47.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:47.403+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:47.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:47.403+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:47.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:47.403+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:47.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:47.404+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:48.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:47.988+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:48.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:47.988+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:48.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:47.988+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:48.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:47.988+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:48.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:48.449+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:48.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:48.449+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:48.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:48.449+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:48.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:48.450+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:48.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:48.371+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:48.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:48.371+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:48.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:48.371+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:48.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:48.372+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:49.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:48.945+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:49.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:48.945+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:49.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:48.945+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:49.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:48.946+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:49.659 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:49.383+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:49.659 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:49.383+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:49.659 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:49.383+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:49.659 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:49.384+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:49.659 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:49.461+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:49.660 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:49.461+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:49.660 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:49.461+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:49.660 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:49.462+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:49.962 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:25:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:25:49.657+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 20496 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:25:50.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:49.960+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:50.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:49.960+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:50.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:49.960+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:50.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:49.961+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:50.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:50.418+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:50.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:50.418+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:50.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:50.418+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:50.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:50.419+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:50.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:50.427+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:50.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:50.427+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:50.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:50.427+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:50.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:50.428+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:51.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:50.998+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:51.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:50.998+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:51.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:50.998+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:51.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:50.998+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:51.718 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:51.451+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:51.719 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:51.451+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:51.719 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:51.451+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:51.719 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:51.452+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:51.719 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:51.461+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:51.719 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:51.461+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:51.719 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:51.461+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:51.719 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:51.463+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:52.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:51.971+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:52.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:51.971+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:52.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:51.971+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:52.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:51.972+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:52.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:52.440+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:52.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:52.440+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:52.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:52.440+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:52.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:52.440+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:52.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:52.492+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:52.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:52.492+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:52.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:52.492+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:52.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:52.492+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:53.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:52.923+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:53.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:52.923+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:53.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:52.923+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:53.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:52.924+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:53.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:53.488+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:53.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:53.488+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:53.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:53.488+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:53.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:53.489+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:53.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:53.450+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:53.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:53.450+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:53.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:53.450+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:53.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:53.451+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:54.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:53.932+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:54.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:53.932+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:54.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:53.932+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:54.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:53.933+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:54.777 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:25:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:25:54.659+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 20573 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:25:54.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:54.495+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:54.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:54.495+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:54.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:54.495+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:54.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:54.496+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:54.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:54.483+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:54.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:54.483+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:54.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:54.483+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:54.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:54.484+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:55.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:54.905+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:55.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:54.905+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:55.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:54.905+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:55.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:54.906+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:55.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:55.441+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:55.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:55.441+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:55.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:55.441+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:55.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:55.441+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:55.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:55.461+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:55.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:55.461+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:55.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:55.461+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:55.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:55.462+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:56.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:55.879+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:56.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:55.879+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:56.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:55.879+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:56.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:55.880+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:56.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:56.428+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:56.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:56.428+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:56.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:56.428+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:56.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:56.429+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:56.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:56.412+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:56.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:56.412+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:56.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:56.412+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:56.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:56.413+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:57.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:56.869+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:57.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:56.869+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:57.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:56.869+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:57.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:56.870+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:57.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:57.425+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:57.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:57.425+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:57.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:57.425+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:57.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:57.426+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:57.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:57.475+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:57.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:57.476+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:57.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:57.476+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:57.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:57.476+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:58.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:57.851+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:58.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:57.851+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:58.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:57.851+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:58.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:57.852+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:58.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:58.490+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:58.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:58.490+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:58.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:58.490+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:58.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:58.491+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:58.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:58.387+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:58.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:58.387+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:58.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:58.387+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:58.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:58.388+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:59.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:58.887+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:59.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:58.887+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:59.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:58.887+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:25:59.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:58.888+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:59.663 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:59.461+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:59.664 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:59.461+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:59.664 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:59.461+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:25:59.664 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:25:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:25:59.461+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:59.664 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:59.424+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:59.664 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:59.424+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:59.664 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:59.424+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:25:59.664 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:25:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:25:59.425+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:25:59.933 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:25:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:25:59.661+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 20655 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:26:00.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:59.931+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:00.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:59.931+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:00.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:59.931+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:00.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:25:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:25:59.932+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:00.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:00.442+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:00.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:00.442+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:00.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:00.442+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:00.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:00.442+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:00.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:00.479+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:00.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:00.479+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:00.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:00.479+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:00.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:00.480+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:01.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:00.930+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:01.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:00.930+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:01.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:00.930+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:01.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:00.931+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:01.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:01.492+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:01.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:01.492+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:01.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:01.492+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:01.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:01.493+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:01.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:01.525+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:01.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:01.525+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:01.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:01.525+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:01.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:01.525+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:02.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:01.942+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:02.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:01.942+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:02.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:01.942+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:02.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:01.943+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:02.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:02.454+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:02.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:02.454+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:02.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:02.454+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:02.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:02.455+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:02.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:02.504+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:02.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:02.504+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:02.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:02.504+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:02.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:02.505+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:03.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:02.978+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:03.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:02.978+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:03.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:02.978+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:03.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:02.979+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:03.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:03.460+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:03.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:03.460+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:03.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:03.460+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:03.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:03.461+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:03.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:03.459+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:03.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:03.459+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:03.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:03.459+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:03.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:03.460+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:04.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:03.931+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:04.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:03.931+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:04.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:03.931+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:04.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:03.932+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:04.777 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:26:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:26:04.663+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 20739 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:26:04.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:04.448+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:04.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:04.448+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:04.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:04.448+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:04.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:04.449+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:04.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:04.476+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:04.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:04.476+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:04.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:04.476+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:04.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:04.477+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:05.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:04.899+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:05.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:04.899+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:05.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:04.899+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:05.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:04.900+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:05.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:05.486+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:05.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:05.486+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:05.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:05.486+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:05.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:05.487+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:05.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:05.424+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:05.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:05.424+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:05.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:05.425+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:05.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:05.425+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:06.243 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:05.913+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:06.244 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:05.913+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:06.244 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:05.913+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:06.244 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:05.914+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:06.512 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:06.443+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:06.512 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:06.443+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:06.512 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:06.443+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:06.513 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:06.444+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:06.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:06.511+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:06.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:06.511+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:06.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:06.511+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:06.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:06.512+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:07.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:06.956+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:07.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:06.956+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:07.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:06.956+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:07.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:06.957+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:07.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:07.401+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:07.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:07.401+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:07.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:07.401+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:07.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:07.402+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:07.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:07.549+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:07.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:07.549+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:07.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:07.549+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:07.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:07.550+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:08.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:07.909+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:08.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:07.909+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:08.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:07.909+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:08.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:07.910+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:08.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:08.595+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:08.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:08.595+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:08.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:08.595+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:08.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:08.596+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:08.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:08.398+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:08.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:08.398+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:08.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:08.398+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:08.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:08.399+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:09.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:08.936+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:09.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:08.936+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:09.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:08.936+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:09.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:08.937+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:09.560 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:09.434+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:09.560 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:09.434+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:09.560 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:09.434+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:09.560 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:09.435+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:09.946 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:26:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:26:09.666+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 20819 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:26:09.946 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:09.558+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:09.947 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:09.558+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:09.947 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:09.558+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:09.947 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:09.559+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:10.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:09.945+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:10.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:09.945+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:10.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:09.945+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:10.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:09.946+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:10.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:10.447+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:10.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:10.447+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:10.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:10.447+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:10.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:10.448+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:10.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:10.568+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:10.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:10.568+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:10.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:10.568+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:10.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:10.569+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:11.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:10.941+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:11.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:10.942+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:11.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:10.942+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:11.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:10.942+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:11.720 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:11.406+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:11.720 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:11.406+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:11.720 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:11.406+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:11.720 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:11.407+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:11.721 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:11.534+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:11.721 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:11.534+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:11.721 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:11.534+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:11.721 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:11.535+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:12.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:11.895+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:12.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:11.895+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:12.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:11.896+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:12.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:11.896+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:12.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:12.537+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:12.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:12.537+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:12.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:12.537+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:12.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:12.538+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:12.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:12.414+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:12.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:12.414+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:12.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:12.414+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:12.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:12.415+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:13.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:12.852+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:13.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:12.852+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:13.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:12.852+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:13.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:12.853+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:13.636 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:13.539+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:13.636 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:13.539+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:13.636 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:13.539+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:13.637 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:13.540+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:13.637 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:13.365+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:13.637 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:13.365+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:13.637 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:13.365+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:13.637 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:13.366+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:14.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:13.866+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:14.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:13.866+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:14.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:13.866+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:14.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:13.866+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:14.670 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:14.512+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:14.670 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:14.512+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:14.670 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:14.512+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:14.670 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:14.513+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:14.671 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:14.327+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:14.671 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:14.327+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:14.671 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:14.327+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:14.671 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:14.328+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:15.027 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:26:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:26:14.668+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 20903 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:26:15.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:14.840+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:15.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:14.840+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:15.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:14.840+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:15.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:14.841+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:15.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:15.522+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:15.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:15.522+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:15.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:15.522+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:15.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:15.523+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:15.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:15.297+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:15.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:15.297+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:15.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:15.297+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:15.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:15.298+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:16.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:15.891+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:16.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:15.891+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:16.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:15.891+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:16.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:15.892+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:16.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:16.515+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:16.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:16.515+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:16.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:16.515+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:16.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:16.516+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:16.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:16.331+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:16.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:16.331+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:16.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:16.331+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:16.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:16.332+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:17.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:16.869+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:17.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:16.870+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:17.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:16.870+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:17.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:16.870+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:17.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:17.495+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:17.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:17.495+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:17.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:17.495+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:17.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:17.496+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:17.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:17.302+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:17.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:17.303+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:17.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:17.303+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:17.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:17.304+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:18.272 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:17.918+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:18.272 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:17.918+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:18.272 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:17.918+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:18.272 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:17.919+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:18.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:18.270+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:18.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:18.270+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:18.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:18.270+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:18.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:18.271+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:18.940 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:18.535+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:18.941 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:18.535+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:18.941 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:18.535+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:18.941 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:18.536+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:19.254 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:18.939+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:19.254 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:18.939+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:19.254 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:18.939+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:19.254 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:18.940+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:19.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:19.252+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:19.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:19.252+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:19.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:19.252+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:19.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:19.253+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:19.799 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:19.530+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:19.799 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:19.530+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:19.799 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:19.530+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:19.799 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:19.531+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:19.799 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:26:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:26:19.670+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 20985 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:26:20.150 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:19.905+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:20.150 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:19.905+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:20.151 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:19.905+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:20.151 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:19.906+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:20.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:20.239+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:20.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:20.239+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:20.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:20.239+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:20.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:20.240+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:20.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:20.524+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:20.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:20.524+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:20.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:20.524+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:20.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:20.525+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:21.269 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:20.881+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:21.269 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:20.881+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:21.269 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:20.881+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:21.270 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:20.882+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:21.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:21.476+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:21.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:21.476+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:21.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:21.476+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:21.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:21.477+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:21.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:21.268+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:21.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:21.268+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:21.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:21.268+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:21.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:21.269+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:22.249 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:21.874+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:22.250 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:21.874+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:22.250 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:21.874+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:22.250 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:21.875+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:22.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:22.248+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:22.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:22.248+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:22.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:22.248+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:22.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:22.249+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:22.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:22.481+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:22.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:22.481+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:22.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:22.481+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:22.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:22.482+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:23.263 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:22.922+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:23.264 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:22.922+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:23.264 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:22.922+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:23.264 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:22.923+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:23.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:23.481+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:23.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:23.481+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:23.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:23.481+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:23.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:23.482+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:23.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:23.262+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:23.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:23.262+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:23.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:23.262+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:23.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:23.263+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:24.265 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:23.953+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:24.265 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:23.953+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:24.265 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:23.953+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:24.265 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:23.954+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:24.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:24.483+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:24.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:24.483+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:24.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:24.483+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:24.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:24.484+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:24.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:24.263+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:24.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:24.263+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:24.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:24.263+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:24.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:24.264+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:24.962 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:26:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:26:24.673+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 21067 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:26:25.236 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:24.960+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:25.236 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:24.960+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:25.236 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:24.960+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:25.237 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:24.961+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:25.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:25.235+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:25.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:25.235+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:25.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:25.235+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:25.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:25.236+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:25.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:25.464+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:25.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:25.464+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:25.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:25.464+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:25.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:25.465+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:26.247 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:25.971+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:26.247 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:25.971+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:26.247 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:25.971+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:26.248 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:25.972+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:26.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:26.246+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:26.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:26.246+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:26.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:26.246+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:26.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:26.247+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:26.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:26.449+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:26.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:26.449+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:26.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:26.449+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:26.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:26.450+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:27.226 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:26.961+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:27.227 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:26.961+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:27.227 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:26.961+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:27.227 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:26.961+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:27.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:27.225+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:27.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:27.225+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:27.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:27.225+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:27.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:27.225+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:27.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:27.450+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:27.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:27.450+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:27.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:27.450+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:27.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:27.450+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:28.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:27.981+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:28.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:27.981+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:28.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:27.981+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:28.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:27.982+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:28.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:28.204+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:28.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:28.204+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:28.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:28.204+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:28.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:28.205+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:28.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:28.411+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:28.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:28.411+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:28.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:28.411+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:28.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:28.412+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:29.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:28.990+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:29.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:28.990+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:29.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:28.990+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:29.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:28.991+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:29.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:29.228+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:29.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:29.228+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:29.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:29.228+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:29.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:29.229+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:29.677 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:29.364+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:29.677 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:29.364+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:29.677 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:29.364+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:29.677 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:29.365+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:29.967 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:26:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:26:29.675+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 21148 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:26:30.259 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:29.966+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:30.260 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:29.966+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:30.260 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:29.966+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:30.260 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:29.967+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:30.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:30.410+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:30.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:30.410+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:30.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:30.410+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:30.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:30.411+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:30.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:30.258+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:30.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:30.258+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:30.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:30.258+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:30.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:30.259+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:31.254 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:30.943+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:31.254 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:30.943+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:31.254 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:30.943+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:31.255 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:30.944+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:31.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:31.426+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:31.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:31.426+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:31.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:31.426+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:31.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:31.427+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:31.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:31.253+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:31.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:31.253+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:31.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:31.253+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:31.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:31.254+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:32.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:31.938+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:32.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:31.938+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:32.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:31.938+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:32.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:31.939+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:32.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:32.425+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:32.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:32.425+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:32.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:32.425+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:32.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:32.426+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:32.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:32.288+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:32.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:32.288+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:32.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:32.288+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:32.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:32.289+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:33.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:32.928+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:33.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:32.928+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:33.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:32.928+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:33.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:32.929+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:33.605 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:33.421+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:33.605 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:33.421+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:33.605 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:33.421+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:33.605 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:33.422+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:33.605 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:33.323+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:33.606 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:33.323+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:33.606 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:33.323+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:33.606 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:33.323+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:34.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:33.973+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:34.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:33.973+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:34.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:33.973+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:34.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:33.973+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:34.679 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:34.461+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:34.679 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:34.461+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:34.680 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:34.461+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:34.680 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:34.462+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:34.680 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:34.311+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:34.680 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:34.311+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:34.681 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:34.311+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:34.681 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:34.312+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:34.963 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:26:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:26:34.677+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 21231 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:26:35.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:34.961+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:35.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:34.961+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:35.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:34.961+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:35.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:34.963+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:35.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:35.490+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:35.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:35.490+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:35.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:35.490+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:35.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:35.491+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:35.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:35.357+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:35.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:35.357+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:35.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:35.357+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:35.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:35.358+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:36.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:36.009+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:36.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:36.009+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:36.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:36.009+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:36.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:36.010+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:36.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:36.375+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:36.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:36.375+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:36.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:36.375+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:36.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:36.375+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:36.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:36.444+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:36.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:36.444+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:36.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:36.444+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:36.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:36.444+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:37.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:36.981+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:37.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:36.981+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:37.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:36.981+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:37.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:36.982+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:37.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:37.415+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:37.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:37.415+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:37.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:37.415+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:37.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:37.415+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:37.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:37.421+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:37.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:37.421+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:37.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:37.421+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:37.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:37.421+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:38.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:38.003+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:38.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:38.003+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:38.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:38.003+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:38.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:38.004+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:38.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:38.436+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:38.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:38.436+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:38.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:38.436+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:38.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:38.437+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:38.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:38.423+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:38.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:38.423+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:38.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:38.423+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:38.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:38.424+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:39.385 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:39.053+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:39.385 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:39.053+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:39.385 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:39.053+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:39.385 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:39.054+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:39.681 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:39.435+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:39.682 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:39.435+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:39.682 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:39.435+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:39.682 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:39.436+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:39.682 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:39.417+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:39.682 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:39.417+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:39.682 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:39.417+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:39.682 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:39.417+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:40.018 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:26:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:26:39.680+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 21313 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:26:40.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:40.017+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:40.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:40.017+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:40.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:40.017+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:40.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:40.018+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:40.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:40.404+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:40.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:40.404+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:40.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:40.404+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:40.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:40.404+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:40.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:40.385+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:40.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:40.385+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:40.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:40.385+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:40.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:40.386+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:41.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:40.996+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:41.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:40.996+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:41.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:40.996+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:41.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:40.997+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:41.722 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:41.351+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:41.722 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:41.351+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:41.722 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:41.351+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:41.722 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:41.352+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:41.723 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:41.384+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:41.723 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:41.384+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:41.723 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:41.384+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:41.723 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:41.385+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:42.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:42.014+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:42.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:42.014+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:42.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:42.014+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:42.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:42.014+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:42.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:42.371+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:42.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:42.371+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:42.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:42.371+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:42.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:42.372+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:42.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:42.371+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:42.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:42.371+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:42.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:42.371+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:42.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:42.373+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:43.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:43.002+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:43.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:43.002+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:43.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:43.002+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:43.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:43.002+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:43.606 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:43.337+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:43.607 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:43.337+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:43.607 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:43.337+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:43.608 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:43.338+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:43.608 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:43.369+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:43.608 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:43.369+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:43.608 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:43.369+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:43.608 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:43.370+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:44.323 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:44.036+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:44.323 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:44.036+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:44.323 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:44.036+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:44.323 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:44.037+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:44.684 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:44.322+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:44.684 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:44.322+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:44.684 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:44.322+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:44.684 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:44.322+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:44.684 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:44.343+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:44.685 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:44.343+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:44.685 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:44.343+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:44.685 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:44.344+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:45.027 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:26:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:26:44.682+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 21397 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:26:45.372 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:45.306+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:45.372 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:45.306+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:45.372 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:45.306+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:45.372 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:45.307+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:45.373 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:45.081+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:45.373 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:45.081+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:45.373 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:45.081+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:45.373 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:45.082+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:45.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:45.370+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:45.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:45.370+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:45.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:45.370+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:45.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:45.371+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:46.314 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:46.053+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:46.315 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:46.053+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:46.315 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:46.053+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:46.315 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:46.054+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:46.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:46.361+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:46.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:46.361+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:46.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:46.361+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:46.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:46.361+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:46.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:46.313+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:46.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:46.313+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:46.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:46.313+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:46.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:46.314+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:47.343 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:47.051+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:47.343 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:47.051+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:47.343 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:47.051+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:47.343 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:47.052+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:47.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:47.341+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:47.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:47.341+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:47.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:47.341+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:47.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:47.342+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:47.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:47.399+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:47.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:47.399+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:47.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:47.399+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:47.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:47.400+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:48.363 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:48.096+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:48.363 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:48.096+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:48.363 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:48.096+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:48.364 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:48.097+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:48.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:48.387+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:48.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:48.387+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:48.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:48.387+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:48.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:48.387+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:48.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:48.362+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:48.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:48.362+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:48.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:48.362+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:48.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:48.363+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:49.322 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:49.048+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:49.323 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:49.048+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:49.323 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:49.048+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:49.323 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:49.050+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:49.686 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:49.395+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:49.686 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:49.395+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:49.686 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:49.395+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:49.687 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:49.396+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:49.687 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:49.321+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:49.687 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:49.321+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:49.687 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:49.321+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:49.687 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:49.322+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:50.027 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:26:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:26:49.684+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 21478 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:26:50.383 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:50.055+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:50.383 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:50.055+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:50.383 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:50.055+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:50.383 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:50.056+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:50.384 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:50.296+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:50.384 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:50.296+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:50.384 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:50.296+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:50.384 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:50.297+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:50.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:50.382+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:50.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:50.382+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:50.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:50.382+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:50.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:50.383+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:51.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:51.007+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:51.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:51.007+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:51.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:51.007+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:51.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:51.007+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:51.722 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:51.338+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:51.722 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:51.338+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:51.722 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:51.338+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:51.723 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:51.339+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:51.723 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:51.415+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:51.723 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:51.415+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:51.723 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:51.415+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:51.723 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:51.416+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:52.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:51.995+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:52.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:51.995+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:52.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:51.995+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:52.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:51.995+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:52.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:52.330+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:52.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:52.330+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:52.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:52.330+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:52.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:52.331+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:52.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:52.396+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:52.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:52.396+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:52.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:52.396+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:52.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:52.397+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:53.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:52.977+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:53.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:52.978+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:53.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:52.978+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:53.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:52.978+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:53.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:53.379+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:53.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:53.379+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:53.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:53.379+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:53.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:53.379+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:53.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:53.428+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:53.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:53.428+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:53.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:53.428+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:53.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:53.429+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:54.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:53.985+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:54.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:53.986+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:54.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:53.986+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:54.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:53.986+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:54.689 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:54.464+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:54.689 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:54.464+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:54.690 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:54.464+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:54.690 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:54.464+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:54.690 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:54.355+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:54.690 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:54.355+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:54.690 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:54.355+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:54.690 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:54.356+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:54.943 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:26:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:26:54.687+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 21561 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:26:55.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:54.942+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:55.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:54.942+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:55.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:54.942+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:55.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:54.943+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:55.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:55.433+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:55.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:55.433+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:55.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:55.433+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:55.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:55.433+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:55.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:55.382+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:55.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:55.382+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:55.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:55.382+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:55.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:55.383+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:56.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:55.900+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:56.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:55.900+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:56.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:55.900+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:56.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:55.901+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:56.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:56.425+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:56.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:56.425+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:56.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:56.425+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:56.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:56.426+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:56.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:56.410+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:56.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:56.410+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:56.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:56.410+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:56.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:56.411+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:57.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:56.914+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:57.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:56.914+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:57.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:56.914+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:57.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:56.915+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:57.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:57.401+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:57.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:57.401+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:57.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:57.401+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:57.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:57.401+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:57.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:57.363+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:57.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:57.363+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:57.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:57.363+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:57.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:57.364+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:58.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:57.921+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:58.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:57.921+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:58.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:57.921+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:58.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:57.922+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:58.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:58.392+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:58.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:58.392+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:58.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:58.392+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:58.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:58.392+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:58.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:58.360+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:58.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:58.360+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:58.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:58.360+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:58.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:58.360+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:59.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:58.920+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:59.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:58.920+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:59.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:58.920+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:26:59.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:58.920+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:59.690 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:59.322+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:59.691 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:59.322+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:59.691 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:59.322+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:26:59.691 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:26:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:26:59.323+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:59.691 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:59.399+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:59.692 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:59.399+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:59.692 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:59.399+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:26:59.692 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:26:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:26:59.400+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:26:59.947 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:26:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:26:59.689+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 21643 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:27:00.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:59.946+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:00.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:59.946+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:00.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:59.946+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:00.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:26:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:26:59.947+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:00.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:00.396+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:00.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:00.396+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:00.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:00.396+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:00.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:00.396+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:00.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:00.360+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:00.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:00.360+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:00.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:00.360+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:00.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:00.361+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:01.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:00.938+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:01.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:00.938+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:01.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:00.938+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:01.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:00.939+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:01.723 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:01.400+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:01.723 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:01.400+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:01.723 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:01.400+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:01.724 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:01.401+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:01.724 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:01.390+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:01.724 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:01.391+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:01.724 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:01.391+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:01.724 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:01.391+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:02.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:01.955+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:02.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:01.955+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:02.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:01.955+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:02.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:01.956+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:02.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:02.425+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:02.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:02.425+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:02.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:02.425+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:02.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:02.426+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:02.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:02.438+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:02.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:02.438+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:02.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:02.438+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:02.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:02.439+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:03.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:02.969+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:03.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:02.969+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:03.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:02.969+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:03.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:02.970+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:03.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:03.423+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:03.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:03.423+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:03.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:03.423+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:03.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:03.423+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:03.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:03.399+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:03.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:03.399+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:03.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:03.399+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:03.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:03.399+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:04.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:04.014+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:04.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:04.014+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:04.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:04.014+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:04.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:04.015+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:04.612 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:04.378+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:04.612 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:04.378+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:04.612 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:04.378+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:04.613 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:04.379+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:04.613 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:04.357+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:04.613 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:04.357+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:04.613 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:04.357+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:04.613 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:04.357+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:04.976 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:27:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:27:04.691+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 21727 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:27:05.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:04.975+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:05.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:04.975+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:05.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:04.975+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:05.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:04.976+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:05.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:05.353+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:05.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:05.353+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:05.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:05.353+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:05.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:05.354+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:05.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:05.339+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:05.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:05.339+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:05.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:05.339+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:05.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:05.340+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:06.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:06.002+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:06.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:06.002+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:06.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:06.002+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:06.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:06.003+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:06.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:06.377+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:06.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:06.377+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:06.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:06.377+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:06.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:06.378+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:06.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:06.341+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:06.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:06.341+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:06.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:06.341+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:06.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:06.342+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:07.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:06.978+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:07.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:06.978+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:07.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:06.978+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:07.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:06.979+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:07.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:07.414+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:07.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:07.414+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:07.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:07.414+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:07.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:07.415+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:07.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:07.384+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:07.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:07.384+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:07.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:07.384+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:07.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:07.384+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:08.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:07.979+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:08.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:07.980+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:08.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:07.980+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:08.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:07.981+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:08.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:08.417+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:08.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:08.417+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:08.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:08.417+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:08.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:08.418+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:08.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:08.365+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:08.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:08.365+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:08.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:08.365+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:08.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:08.366+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:09.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:08.968+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:09.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:08.968+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:09.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:08.968+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:09.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:08.969+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:09.695 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:09.353+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:09.695 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:09.353+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:09.695 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:09.353+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:09.695 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:09.354+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:09.696 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:09.433+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:09.696 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:09.433+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:09.696 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:09.433+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:09.696 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:09.434+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:10.027 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:27:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:27:09.694+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 21809 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:27:10.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:09.942+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:10.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:09.942+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:10.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:09.942+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:10.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:09.943+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:10.477 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:10.336+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:10.477 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:10.336+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:10.478 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:10.336+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:10.478 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:10.336+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:10.755 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:10.476+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:10.756 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:10.476+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:10.756 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:10.476+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:10.756 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:10.477+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:11.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:10.945+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:11.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:10.945+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:11.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:10.945+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:11.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:10.945+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:11.723 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:11.462+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:11.723 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:11.462+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:11.724 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:11.462+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:11.724 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:11.463+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:11.724 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:11.381+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:11.724 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:11.381+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:11.724 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:11.381+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:11.724 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:11.382+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:12.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:11.978+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:12.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:11.978+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:12.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:11.978+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:12.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:11.979+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:12.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:12.339+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:12.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:12.339+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:12.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:12.339+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:12.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:12.340+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:12.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:12.486+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:12.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:12.486+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:12.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:12.486+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:12.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:12.487+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:13.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:13.012+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:13.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:13.012+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:13.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:13.012+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:13.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:13.013+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:13.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:13.467+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:13.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:13.467+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:13.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:13.467+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:13.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:13.468+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:13.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:13.388+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:13.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:13.388+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:13.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:13.388+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:13.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:13.389+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:14.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:13.965+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:14.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:13.965+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:14.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:13.965+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:14.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:13.966+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:14.699 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:14.421+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:14.699 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:14.421+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:14.699 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:14.421+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:14.699 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:14.422+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:14.699 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:14.492+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:14.699 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:14.492+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:14.699 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:14.492+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:14.700 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:14.492+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:15.027 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:27:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:27:14.696+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 21893 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:27:15.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:14.942+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:15.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:14.942+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:15.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:14.942+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:15.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:14.942+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:15.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:15.477+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:15.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:15.477+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:15.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:15.477+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:15.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:15.478+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:15.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:15.437+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:15.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:15.437+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:15.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:15.437+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:15.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:15.438+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:16.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:15.898+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:16.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:15.898+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:16.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:15.898+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:16.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:15.899+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:16.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:16.443+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:16.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:16.443+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:16.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:16.443+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:16.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:16.444+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:16.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:16.441+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:16.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:16.441+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:16.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:16.441+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:16.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:16.442+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:17.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:16.899+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:17.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:16.899+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:17.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:16.899+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:17.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:16.900+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:17.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:17.402+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:17.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:17.402+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:17.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:17.402+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:17.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:17.403+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:17.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:17.395+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:17.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:17.395+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:17.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:17.395+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:17.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:17.396+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:18.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:17.919+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:18.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:17.919+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:18.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:17.919+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:18.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:17.920+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:18.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:18.374+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:18.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:18.374+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:18.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:18.374+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:18.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:18.375+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:18.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:18.422+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:18.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:18.422+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:18.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:18.422+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:18.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:18.422+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:19.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:18.969+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:19.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:18.969+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:19.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:18.969+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:19.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:18.970+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:19.700 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:19.400+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:19.700 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:19.400+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:19.700 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:19.400+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:19.700 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:19.401+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:19.700 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:19.333+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:19.701 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:19.333+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:19.701 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:19.333+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:19.701 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:19.334+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:19.982 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:27:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:27:19.698+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 21975 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:27:20.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:19.981+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:20.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:19.981+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:20.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:19.981+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:20.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:19.982+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:20.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:20.438+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:20.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:20.438+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:20.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:20.438+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:20.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:20.439+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:20.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:20.312+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:20.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:20.312+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:20.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:20.312+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:20.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:20.313+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:21.269 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:20.974+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:21.269 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:20.974+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:21.269 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:20.974+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:21.270 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:20.975+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:21.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:21.428+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:21.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:21.428+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:21.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:21.428+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:21.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:21.429+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:21.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:21.268+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:21.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:21.268+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:21.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:21.268+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:21.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:21.269+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:22.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:22.224+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:22.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:22.224+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:22.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:22.224+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:22.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:22.225+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:22.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:22.002+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:22.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:22.002+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:22.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:22.002+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:22.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:22.003+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:22.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:22.425+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:22.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:22.425+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:22.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:22.425+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:22.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:22.426+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:23.421 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:23.242+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:23.421 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:23.242+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:23.421 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:23.242+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:23.421 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:23.242+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:23.421 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:23.047+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:23.422 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:23.047+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:23.422 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:23.047+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:23.422 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:23.048+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:23.422 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:23.419+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:23.422 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:23.419+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:23.422 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:23.419+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:23.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:23.420+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:24.271 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:24.015+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:24.271 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:24.015+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:24.271 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:24.015+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:24.271 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:24.016+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:24.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:24.388+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:24.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:24.388+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:24.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:24.388+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:24.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:24.389+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:24.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:24.270+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:24.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:24.270+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:24.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:24.270+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:24.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:24.271+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:25.027 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:27:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:27:24.701+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 22059 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:27:25.286 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:25.034+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:25.287 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:25.034+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:25.287 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:25.034+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:25.287 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:25.034+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:25.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:25.285+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:25.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:25.285+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:25.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:25.285+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:25.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:25.286+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:25.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:25.415+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:25.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:25.415+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:25.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:25.415+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:25.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:25.415+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:26.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:26.015+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:26.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:26.015+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:26.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:26.015+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:26.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:26.016+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:26.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:26.306+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:26.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:26.306+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:26.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:26.306+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:26.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:26.307+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:26.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:26.400+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:26.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:26.400+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:26.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:26.400+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:26.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:26.401+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:27.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:27.002+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:27.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:27.002+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:27.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:27.002+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:27.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:27.003+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:27.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:27.320+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:27.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:27.320+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:27.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:27.320+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:27.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:27.320+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:27.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:27.368+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:27.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:27.368+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:27.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:27.368+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:27.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:27.368+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:28.338 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:28.039+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:28.338 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:28.039+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:28.338 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:28.039+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:28.338 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:28.040+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:28.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:28.409+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:28.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:28.409+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:28.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:28.409+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:28.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:28.410+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:28.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:28.337+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:28.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:28.337+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:28.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:28.337+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:28.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:28.338+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:29.309 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:29.043+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:29.310 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:29.043+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:29.310 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:29.043+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:29.310 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:29.044+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:29.704 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:29.308+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:29.705 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:29.308+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:29.705 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:29.308+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:29.705 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:29.309+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:29.705 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:29.364+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:29.706 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:29.364+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:29.706 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:29.365+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:29.706 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:29.366+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:30.027 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:27:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:27:29.703+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 22141 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:27:30.292 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:30.028+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:30.292 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:30.028+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:30.292 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:30.028+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:30.293 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:30.029+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:30.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:30.407+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:30.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:30.407+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:30.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:30.407+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:30.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:30.408+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:30.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:30.291+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:30.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:30.291+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:30.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:30.291+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:30.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:30.292+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:31.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:31.250+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:31.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:31.250+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:31.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:31.250+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:31.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:31.251+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:31.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:31.009+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:31.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:31.009+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:31.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:31.009+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:31.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:31.009+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:31.725 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:31.456+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:31.725 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:31.456+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:31.725 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:31.456+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:31.725 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:31.457+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:32.312 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:32.028+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:32.312 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:32.028+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:32.312 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:32.028+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:32.312 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:32.029+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:32.312 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:32.202+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:32.313 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:32.202+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:32.313 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:32.202+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:32.313 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:32.203+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:32.649 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:32.473+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:32.649 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:32.473+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:32.650 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:32.473+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:32.650 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:32.474+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:33.261 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:33.199+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:33.262 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:33.199+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:33.262 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:33.199+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:33.262 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:33.200+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:33.262 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:32.996+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:33.262 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:32.996+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:33.263 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:32.996+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:33.263 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:32.997+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:33.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:33.438+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:33.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:33.438+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:33.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:33.438+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:33.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:33.439+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:34.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:34.186+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:34.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:34.186+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:34.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:34.186+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:34.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:34.186+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:34.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:34.004+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:34.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:34.004+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:34.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:34.004+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:34.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:34.005+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:34.730 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:27:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:27:34.705+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 22224 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:27:34.731 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:34.475+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:34.731 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:34.475+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:34.731 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:34.475+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:34.731 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:34.476+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:35.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:34.996+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:35.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:34.996+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:35.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:34.996+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:35.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:34.997+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:35.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:35.194+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:35.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:35.194+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:35.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:35.194+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:35.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:35.195+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:35.691 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:35.485+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:35.691 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:35.485+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:35.691 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:35.485+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:35.691 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:35.486+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:36.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:35.968+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:36.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:35.968+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:36.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:35.968+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:36.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:35.969+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:36.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:36.202+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:36.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:36.202+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:36.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:36.202+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:36.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:36.203+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:36.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:36.490+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:36.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:36.490+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:36.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:36.490+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:36.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:36.491+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:37.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:36.999+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:37.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:36.999+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:37.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:36.999+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:37.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:37.000+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:37.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:37.246+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:37.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:37.246+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:37.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:37.246+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:37.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:37.247+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:37.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:37.466+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:37.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:37.466+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:37.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:37.466+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:37.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:37.467+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:38.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:37.980+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:38.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:37.980+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:38.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:37.980+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:38.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:37.980+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:38.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:38.476+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:38.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:38.476+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:38.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:38.476+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:38.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:38.478+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:38.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:38.283+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:38.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:38.283+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:38.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:38.283+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:38.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:38.284+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:39.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:38.992+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:39.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:38.992+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:39.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:38.992+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:39.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:38.992+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:39.710 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:39.317+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:39.710 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:39.317+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:39.710 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:39.317+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:39.710 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:39.318+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:39.711 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:39.479+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:39.711 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:39.479+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:39.711 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:39.479+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:39.711 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:39.480+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:40.014 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:27:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:27:39.708+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 22307 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:27:40.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:40.013+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:40.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:40.013+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:40.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:40.013+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:40.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:40.014+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:40.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:40.497+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:40.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:40.497+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:40.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:40.497+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:40.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:40.498+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:40.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:40.295+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:40.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:40.295+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:40.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:40.295+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:40.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:40.295+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:41.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:40.980+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:41.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:40.980+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:41.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:40.980+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:41.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:40.981+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:41.727 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:41.528+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:41.727 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:41.528+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:41.727 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:41.528+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:41.728 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:41.529+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:41.728 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:41.335+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:41.728 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:41.335+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:41.728 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:41.335+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:41.729 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:41.335+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:42.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:41.970+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:42.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:41.970+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:42.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:41.970+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:42.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:41.971+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:42.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:42.376+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:42.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:42.376+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:42.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:42.376+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:42.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:42.377+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:42.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:42.503+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:42.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:42.503+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:42.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:42.503+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:42.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:42.505+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:43.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:43.012+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:43.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:43.012+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:43.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:43.012+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:43.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:43.012+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:43.533 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:43.469+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:43.533 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:43.469+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:43.533 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:43.469+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:43.534 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:43.469+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:43.534 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:43.335+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:43.534 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:43.335+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:43.534 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:43.335+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:43.534 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:43.336+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:44.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:44.030+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:44.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:44.030+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:44.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:44.030+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:44.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:44.031+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:44.713 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:44.306+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:44.713 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:44.306+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:44.713 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:44.306+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:44.713 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:44.307+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:44.714 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:44.421+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:44.714 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:44.421+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:44.714 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:44.421+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:44.714 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:44.422+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:44.990 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:27:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:27:44.710+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 22390 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:27:45.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:44.989+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:45.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:44.989+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:45.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:44.989+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:45.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:44.990+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:45.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:45.349+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:45.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:45.349+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:45.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:45.349+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:45.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:45.350+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:45.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:45.429+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:45.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:45.429+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:45.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:45.429+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:45.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:45.429+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:46.257 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:45.966+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:46.257 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:45.966+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:46.257 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:45.966+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:46.257 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:45.966+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:46.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:46.439+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:46.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:46.439+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:46.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:46.439+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:46.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:46.440+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:46.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:46.344+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:46.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:46.344+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:46.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:46.344+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:46.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:46.345+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:47.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:46.953+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:47.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:46.953+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:47.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:46.953+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:47.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:46.955+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:47.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:47.346+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:47.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:47.346+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:47.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:47.346+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:47.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:47.347+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:47.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:47.467+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:47.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:47.467+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:47.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:47.467+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:47.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:47.468+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:48.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:47.968+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:48.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:47.969+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:48.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:47.969+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:48.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:47.969+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:48.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:48.505+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:48.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:48.505+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:48.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:48.505+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:48.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:48.506+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:48.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:48.384+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:48.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:48.384+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:48.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:48.384+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:48.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:48.385+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:49.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:49.018+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:49.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:49.018+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:49.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:49.018+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:49.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:49.018+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:49.715 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:49.346+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:49.716 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:49.346+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:49.716 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:49.346+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:49.716 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:49.347+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:49.716 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:49.543+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:49.716 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:49.543+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:49.716 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:49.543+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:49.716 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:49.544+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:49.991 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:27:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:27:49.713+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 22472 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:27:50.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:49.990+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:50.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:49.990+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:50.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:49.990+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:50.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:49.991+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:50.691 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:50.372+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:50.692 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:50.372+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:50.692 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:50.372+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:50.692 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:50.373+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:50.692 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:50.580+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:50.692 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:50.580+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:50.692 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:50.580+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:50.693 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:50.581+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:51.386 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:51.031+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:51.386 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:51.031+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:51.387 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:51.031+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:51.387 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:51.032+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:51.725 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:51.616+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:51.725 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:51.616+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:51.726 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:51.616+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:51.726 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:51.617+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:51.726 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:51.385+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:51.726 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:51.385+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:51.726 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:51.385+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:51.727 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:51.386+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:52.423 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:52.035+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:52.423 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:52.035+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:52.423 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:52.035+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:52.423 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:52.037+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:52.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:52.619+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:52.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:52.619+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:52.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:52.619+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:52.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:52.619+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:52.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:52.423+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:52.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:52.423+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:52.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:52.423+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:52.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:52.423+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:53.440 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:53.050+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:53.440 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:53.050+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:53.441 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:53.050+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:53.441 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:53.051+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:53.763 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:53.578+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:53.763 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:53.578+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:53.764 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:53.578+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:53.764 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:53.579+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:53.764 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:53.439+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:53.764 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:53.439+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:53.764 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:53.439+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:53.765 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:53.440+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:54.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:54.018+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:54.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:54.018+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:54.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:54.018+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:54.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:54.019+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:54.726 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:27:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:27:54.715+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 22554 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:27:54.726 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:54.582+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:54.726 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:54.582+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:54.727 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:54.582+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:54.727 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:54.583+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:54.727 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:54.474+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:54.727 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:54.474+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:54.727 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:54.474+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:54.727 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:54.475+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:55.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:54.989+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:55.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:54.989+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:55.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:54.989+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:55.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:54.989+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:55.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:55.458+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:55.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:55.458+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:55.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:55.458+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:55.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:55.459+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:55.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:55.544+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:55.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:55.544+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:55.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:55.544+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:55.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:55.544+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:56.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:55.953+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:56.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:55.953+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:56.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:55.953+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:56.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:55.953+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:56.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:56.413+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:56.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:56.413+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:56.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:56.413+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:56.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:56.413+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:56.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:56.579+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:56.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:56.579+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:56.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:56.579+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:56.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:56.579+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:57.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:56.974+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:57.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:56.974+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:57.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:56.974+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:57.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:56.975+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:57.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:57.604+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:57.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:57.604+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:57.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:57.604+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:57.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:57.605+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:57.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:57.431+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:57.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:57.431+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:57.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:57.431+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:57.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:57.431+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:58.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:58.016+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:58.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:58.016+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:58.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:58.016+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:58.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:58.017+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:58.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:58.591+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:58.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:58.591+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:58.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:58.591+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:27:58.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:58.592+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:58.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:58.475+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:58.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:58.475+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:58.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:58.475+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:58.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:58.476+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:59.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:58.992+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:59.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:58.992+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:59.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:58.992+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:27:59.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:27:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:27:58.993+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:27:59.640 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:59.434+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:59.641 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:59.434+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:59.641 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:59.434+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:27:59.641 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:27:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:27:59.435+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:00.027 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:27:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:27:59.718+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 22637 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:28:00.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:59.639+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:00.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:59.639+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:00.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:59.639+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:00.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:27:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:27:59.640+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:00.417 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:00.032+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:00.418 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:00.032+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:00.418 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:00.032+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:00.418 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:00.033+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:00.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:00.591+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:00.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:00.591+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:00.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:00.591+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:00.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:00.592+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:00.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:00.416+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:00.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:00.416+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:00.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:00.416+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:00.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:00.417+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:01.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:01.008+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:01.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:01.008+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:01.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:01.008+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:01.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:01.009+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:01.725 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:01.586+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:01.726 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:01.586+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:01.726 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:01.586+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:01.726 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:01.587+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:01.726 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:01.420+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:01.726 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:01.420+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:01.726 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:01.420+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:01.727 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:01.421+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:02.446 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:02.044+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:02.447 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:02.044+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:02.447 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:02.044+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:02.447 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:02.045+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:02.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:02.446+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:02.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:02.446+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:02.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:02.446+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:02.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:02.447+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:02.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:02.578+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:02.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:02.578+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:02.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:02.578+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:02.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:02.578+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:03.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:03.092+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:03.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:03.092+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:03.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:03.092+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:03.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:03.093+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:03.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:03.411+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:03.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:03.411+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:03.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:03.411+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:03.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:03.413+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:03.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:03.606+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:03.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:03.606+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:03.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:03.606+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:03.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:03.607+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:04.428 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:04.113+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:04.428 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:04.113+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:04.428 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:04.113+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:04.428 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:04.113+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:04.723 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:04.427+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:04.723 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:04.427+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:04.723 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:04.427+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:04.723 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:04.428+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:04.723 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:04.594+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:04.724 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:04.594+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:04.724 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:04.594+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:04.724 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:04.594+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:05.027 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:28:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:28:04.720+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 22720 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:28:05.430 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:05.067+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:05.430 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:05.067+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:05.431 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:05.067+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:05.431 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:05.067+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:05.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:05.429+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:05.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:05.429+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:05.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:05.429+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:05.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:05.430+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:05.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:05.640+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:05.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:05.640+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:05.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:05.640+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:05.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:05.641+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:06.413 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:06.089+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:06.413 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:06.089+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:06.413 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:06.089+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:06.413 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:06.090+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:06.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:06.412+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:06.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:06.412+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:06.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:06.412+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:06.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:06.413+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:06.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:06.651+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:06.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:06.651+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:06.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:06.651+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:06.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:06.652+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:07.395 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:07.056+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:07.395 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:07.056+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:07.395 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:07.056+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:07.395 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:07.057+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:07.701 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:07.394+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:07.701 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:07.394+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:07.701 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:07.394+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:07.701 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:07.395+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:08.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:07.700+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:08.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:07.700+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:08.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:07.700+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:08.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:07.701+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:08.366 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:08.068+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:08.367 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:08.068+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:08.367 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:08.068+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:08.367 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:08.069+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:08.663 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:08.365+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:08.664 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:08.365+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:08.664 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:08.365+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:08.664 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:08.366+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:09.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:08.662+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:09.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:08.662+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:09.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:08.662+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:09.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:08.663+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:09.385 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:09.119+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:09.385 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:09.119+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:09.385 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:09.119+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:09.385 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:09.119+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:09.657 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:09.416+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:09.658 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:09.416+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:09.658 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:09.416+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:09.658 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:09.417+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:10.026 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:28:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:28:09.723+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 22803 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:28:10.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:09.657+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:10.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:09.657+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:10.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:09.657+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:10.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:09.657+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:10.374 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:10.083+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:10.375 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:10.083+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:10.375 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:10.083+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:10.375 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:10.084+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:10.667 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:10.373+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:10.667 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:10.373+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:10.667 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:10.373+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:10.667 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:10.375+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:11.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:10.666+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:11.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:10.666+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:11.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:10.666+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:11.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:10.667+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:11.349 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:11.058+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:11.349 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:11.058+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:11.349 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:11.058+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:11.349 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:11.059+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:11.664 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:11.348+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:11.664 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:11.348+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:11.664 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:11.348+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:11.664 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:11.348+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:12.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:11.663+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:12.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:11.663+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:12.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:11.663+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:12.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:11.663+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:12.374 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:12.108+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:12.374 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:12.108+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:12.374 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:12.108+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:12.374 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:12.108+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:12.713 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:12.373+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:12.714 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:12.373+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:12.714 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:12.373+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:12.714 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:12.373+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:13.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:12.712+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:13.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:12.712+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:13.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:12.712+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:13.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:12.714+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:13.398 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:13.107+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:13.398 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:13.107+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:13.398 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:13.107+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:13.398 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:13.108+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:13.707 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:13.397+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:13.708 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:13.397+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:13.708 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:13.397+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:13.708 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:13.398+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:14.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:13.707+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:14.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:13.707+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:14.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:13.707+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:14.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:13.707+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:14.446 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:14.126+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:14.447 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:14.126+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:14.447 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:14.126+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:14.447 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:14.127+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:14.727 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:14.445+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:14.727 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:14.445+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:14.727 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:14.445+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:14.727 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:14.446+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:15.027 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:28:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:28:14.726+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 22896 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:28:15.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:14.737+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:15.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:14.737+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:15.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:14.737+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:15.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:14.738+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:15.490 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:15.078+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:15.490 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:15.078+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:15.490 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:15.078+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:15.490 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:15.079+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:15.742 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:15.489+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:15.742 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:15.489+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:15.742 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:15.489+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:15.742 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:15.490+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:16.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:15.741+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:16.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:15.741+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:16.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:15.741+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:16.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:15.742+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:16.449 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:16.046+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:16.449 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:16.046+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:16.449 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:16.046+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:16.449 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:16.047+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:16.735 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:16.448+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:16.735 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:16.448+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:16.736 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:16.448+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:16.736 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:16.449+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:17.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:16.734+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:17.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:16.734+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:17.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:16.734+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:17.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:16.735+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:17.460 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:17.052+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:17.460 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:17.052+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:17.460 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:17.052+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:17.460 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:17.053+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:17.721 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:17.459+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:17.721 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:17.459+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:17.721 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:17.459+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:17.721 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:17.460+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:18.010 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:17.720+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:18.010 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:17.720+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:18.010 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:17.720+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:18.010 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:17.721+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:18.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:18.009+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:18.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:18.009+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:18.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:18.009+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:18.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:18.010+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:18.716 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:18.419+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:18.716 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:18.419+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:18.716 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:18.419+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:18.717 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:18.420+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:18.989 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:18.715+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:18.989 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:18.715+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:18.990 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:18.715+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:18.990 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:18.716+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:19.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:18.988+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:19.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:18.988+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:19.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:18.988+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:19.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:18.989+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:19.716 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:19.410+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:19.717 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:19.410+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:19.717 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:19.410+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:19.717 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:19.411+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:20.026 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:28:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:28:19.728+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 22979 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:28:20.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:19.715+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:20.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:19.715+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:20.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:19.715+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:20.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:19.716+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:20.403 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:20.035+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:20.404 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:20.035+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:20.404 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:20.035+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:20.404 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:20.036+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:20.685 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:20.402+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:20.685 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:20.402+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:20.686 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:20.402+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:20.686 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:20.403+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:20.999 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:20.684+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:20.999 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:20.684+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:20.999 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:20.684+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:20.999 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:20.685+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:21.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:20.998+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:21.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:20.998+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:21.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:20.998+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:21.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:20.999+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:21.710 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:21.444+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:21.711 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:21.444+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:21.711 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:21.444+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:21.711 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:21.445+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:22.008 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:21.709+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:22.009 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:21.709+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:22.009 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:21.709+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:22.009 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:21.710+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:22.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:22.007+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:22.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:22.007+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:22.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:22.007+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:22.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:22.009+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:22.752 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:22.414+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:22.753 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:22.414+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:22.753 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:22.414+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:22.753 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:22.415+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:23.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:22.751+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:23.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:22.751+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:23.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:22.751+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:23.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:22.752+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:23.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:22.985+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:23.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:22.985+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:23.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:22.985+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:23.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:22.986+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:23.712 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:23.456+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:23.713 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:23.456+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:23.713 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:23.456+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:23.713 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:23.457+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:23.977 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:23.712+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:23.977 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:23.712+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:23.977 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:23.712+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:23.978 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:23.712+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:24.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:23.976+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:24.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:23.976+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:24.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:23.976+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:24.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:23.977+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:24.732 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:24.700+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:24.732 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:24.700+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:24.732 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:24.700+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:24.733 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:24.700+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:24.733 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:24.477+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:24.733 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:24.477+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:24.733 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:24.477+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:24.733 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:24.478+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:24.983 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:28:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:28:24.731+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 23062 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:28:25.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:24.983+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:25.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:24.983+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:25.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:24.983+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:25.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:24.984+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:25.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:25.706+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:25.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:25.706+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:25.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:25.706+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:25.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:25.707+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:25.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:25.508+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:25.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:25.508+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:25.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:25.508+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:25.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:25.509+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:26.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:25.975+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:26.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:25.975+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:26.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:25.975+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:26.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:25.976+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:26.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:26.753+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:26.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:26.753+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:26.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:26.753+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:26.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:26.754+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:26.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:26.507+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:26.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:26.507+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:26.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:26.507+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:26.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:26.508+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:27.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:26.966+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:27.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:26.966+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:27.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:26.966+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:27.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:26.967+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:27.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:27.752+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:27.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:27.752+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:27.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:27.752+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:27.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:27.753+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:27.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:27.524+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:27.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:27.524+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:27.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:27.524+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:27.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:27.525+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:28.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:28.002+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:28.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:28.002+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:28.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:28.002+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:28.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:28.003+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:28.972 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:28.778+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:28.973 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:28.778+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:28.973 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:28.778+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:28.973 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:28.778+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:28.973 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:28.542+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:28.973 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:28.542+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:28.974 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:28.542+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:28.974 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:28.543+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:29.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:28.971+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:29.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:28.971+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:29.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:28.971+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:29.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:28.972+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:29.735 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:29.544+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:29.735 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:29.544+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:29.735 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:29.544+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:29.735 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:29.545+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:30.026 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:28:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:28:29.734+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 23145 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:28:30.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:29.950+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:30.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:29.950+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:30.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:29.950+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:30.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:29.951+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:30.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:29.802+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:30.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:29.802+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:30.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:29.802+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:30.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:29.803+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:30.799 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:30.547+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:30.800 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:30.547+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:30.800 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:30.547+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:30.800 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:30.548+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:31.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:30.798+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:31.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:30.798+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:31.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:30.798+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:31.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:30.799+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:31.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:30.958+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:31.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:30.958+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:31.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:30.958+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:31.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:30.958+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:31.830 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:31.549+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:31.830 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:31.549+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:31.830 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:31.549+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:31.830 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:31.550+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:32.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:31.957+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:32.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:31.957+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:32.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:31.957+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:32.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:31.957+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:32.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:31.829+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:32.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:31.829+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:32.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:31.829+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:32.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:31.830+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:32.828 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:32.559+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:32.828 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:32.559+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:32.828 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:32.559+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:32.829 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:32.560+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:33.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:32.827+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:33.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:32.827+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:33.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:32.827+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:33.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:32.828+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:33.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:32.938+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:33.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:32.938+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:33.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:32.938+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:33.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:32.938+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:33.863 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:33.600+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:33.864 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:33.600+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:33.864 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:33.600+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:33.864 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:33.600+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:34.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:33.862+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:34.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:33.862+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:34.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:33.862+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:34.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:33.863+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:34.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:33.943+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:34.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:33.943+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:34.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:33.943+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:34.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:33.944+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:34.915 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:28:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:28:34.736+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 23228 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:28:34.916 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:34.868+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:34.916 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:34.868+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:34.916 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:34.868+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:34.916 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:34.868+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:34.916 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:34.643+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:34.917 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:34.643+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:34.917 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:34.643+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:34.917 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:34.644+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:35.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:34.914+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:35.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:34.914+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:35.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:34.914+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:35.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:34.916+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:36.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:35.874+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:36.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:35.874+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:36.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:35.874+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:36.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:35.875+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:36.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:35.644+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:36.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:35.644+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:36.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:35.644+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:36.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:35.646+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:36.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:35.851+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:36.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:35.851+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:36.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:35.851+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:36.029 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:35.852+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:37.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:36.829+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:37.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:36.829+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:37.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:36.829+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:37.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:36.830+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:37.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:36.900+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:37.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:36.900+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:37.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:36.900+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:37.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:36.900+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:37.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:36.680+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:37.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:36.680+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:37.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:36.680+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:37.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:36.681+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:37.924 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:37.649+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:37.925 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:37.649+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:37.925 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:37.649+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:37.925 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:37.650+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:37.925 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:37.849+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:37.925 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:37.849+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:37.926 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:37.849+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:37.926 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:37.850+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:38.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:37.923+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:38.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:37.923+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:38.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:37.923+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:38.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:37.924+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:38.953 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:38.692+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:38.954 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:38.692+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:38.954 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:38.692+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:38.954 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:38.693+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:38.954 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:38.892+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:38.955 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:38.892+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:38.955 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:38.892+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:38.955 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:38.893+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:39.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:38.953+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:39.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:38.953+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:39.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:38.953+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:39.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:38.953+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:40.026 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:28:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:28:39.739+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 23309 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:28:40.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:39.707+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:40.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:39.707+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:40.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:39.707+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:40.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:39.708+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:40.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:39.917+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:40.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:39.917+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:40.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:39.917+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:40.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:39.918+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:40.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:39.904+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:40.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:39.904+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:40.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:39.904+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:40.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:39.905+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:41.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:40.928+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:41.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:40.928+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:41.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:40.928+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:41.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:40.929+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:41.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:40.889+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:41.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:40.889+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:41.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:40.889+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:41.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:40.890+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:41.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:40.735+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:41.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:40.735+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:41.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:40.735+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:41.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:40.736+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:42.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:41.902+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:42.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:41.902+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:42.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:41.902+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:42.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:41.903+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:42.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:41.899+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:42.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:41.899+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:42.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:41.899+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:42.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:41.900+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:42.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:41.778+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:42.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:41.778+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:42.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:41.778+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:42.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:41.778+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:43.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:42.932+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:43.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:42.932+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:43.029 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:42.932+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:43.029 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:42.933+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:43.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:42.932+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:43.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:42.932+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:43.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:42.932+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:43.030 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:42.933+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:43.030 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:42.771+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:43.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:42.771+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:43.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:42.771+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:43.031 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:42.771+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:43.890 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:43.768+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:43.890 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:43.768+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:43.890 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:43.768+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:43.890 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:43.768+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:44.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:43.889+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:44.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:43.889+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:44.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:43.889+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:44.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:43.890+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:44.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:43.976+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:44.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:43.976+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:44.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:43.976+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:44.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:43.977+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:45.026 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:28:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:28:44.742+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 23391 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:28:45.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:44.867+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:45.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:44.867+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:45.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:44.867+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:45.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:44.868+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:45.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:44.943+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:45.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:44.943+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:45.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:44.943+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:45.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:44.944+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:45.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:44.722+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:45.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:44.722+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:45.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:44.722+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:45.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:44.723+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:46.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:45.829+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:46.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:45.829+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:46.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:45.829+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:46.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:45.830+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:46.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:45.742+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:46.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:45.742+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:46.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:45.742+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:46.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:45.742+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:46.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:45.900+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:46.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:45.900+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:46.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:45.900+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:46.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:45.901+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:47.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:46.846+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:47.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:46.846+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:47.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:46.846+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:47.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:46.847+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:47.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:46.860+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:47.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:46.860+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:47.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:46.860+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:47.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:46.861+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:47.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:46.786+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:47.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:46.786+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:47.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:46.786+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:47.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:46.787+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:48.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:47.844+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:48.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:47.844+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:48.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:47.844+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:48.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:47.845+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:48.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:47.780+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:48.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:47.780+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:48.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:47.780+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:48.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:47.781+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:48.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:47.858+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:48.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:47.858+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:48.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:47.858+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:48.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:47.859+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:49.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:48.847+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:49.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:48.847+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:49.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:48.847+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:49.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:48.848+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:49.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:48.744+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:49.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:48.744+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:49.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:48.744+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:49.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:48.745+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:49.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:48.904+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:49.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:48.904+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:49.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:48.904+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:49.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:48.905+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:50.026 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:28:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:28:49.744+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 23474 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:28:50.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:49.728+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:50.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:49.728+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:50.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:49.728+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:50.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:49.729+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:50.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:49.867+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:50.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:49.867+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:50.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:49.867+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:50.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:49.868+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:50.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:49.866+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:50.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:49.866+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:50.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:49.866+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:50.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:49.867+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:51.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:50.834+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:51.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:50.834+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:51.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:50.834+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:51.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:50.835+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:51.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:50.745+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:51.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:50.745+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:51.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:50.745+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:51.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:50.746+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:51.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:50.838+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:51.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:50.838+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:51.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:50.838+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:51.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:50.839+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:52.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:51.847+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:52.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:51.847+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:52.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:51.847+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:52.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:51.848+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:52.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:51.746+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:52.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:51.746+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:52.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:51.746+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:52.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:51.747+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:52.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:51.878+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:52.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:51.878+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:52.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:51.878+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:52.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:51.879+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:53.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:52.878+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:53.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:52.878+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:53.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:52.878+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:53.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:52.878+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:53.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:52.730+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:53.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:52.730+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:53.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:52.730+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:53.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:52.730+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:53.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:52.912+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:53.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:52.912+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:53.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:52.912+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:53.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:52.912+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:53.925 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:53.760+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:53.926 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:53.760+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:53.926 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:53.760+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:53.926 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:53.761+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:54.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:53.925+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:54.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:53.925+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:54.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:53.925+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:54.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:53.925+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:54.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:53.929+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:54.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:53.929+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:54.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:53.929+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:54.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:53.930+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:55.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:54.944+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:55.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:54.944+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:55.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:54.944+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:55.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:54.944+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:55.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:54.778+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:55.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:54.778+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:55.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:54.778+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:55.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:54.778+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:55.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:54.970+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:55.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:54.970+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:55.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:54.970+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:55.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:54.971+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:55.029 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:28:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:28:54.747+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 23557 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:28:56.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:55.755+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:56.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:55.755+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:56.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:55.755+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:56.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:55.756+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:56.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:55.950+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:56.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:55.950+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:56.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:55.950+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:56.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:55.951+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:56.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:55.959+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:56.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:55.959+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:56.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:55.959+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:56.029 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:55.959+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:57.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:56.927+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:57.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:56.927+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:57.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:56.927+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:57.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:56.928+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:57.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:56.794+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:57.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:56.794+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:57.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:56.794+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:57.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:56.795+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:57.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:56.996+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:57.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:56.996+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:57.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:56.996+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:57.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:56.997+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:58.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:57.963+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:58.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:57.963+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:58.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:57.963+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:58.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:57.964+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:58.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:57.794+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:58.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:57.794+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:58.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:57.794+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:58.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:57.795+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:58.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:57.990+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:58.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:57.990+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:58.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:57.990+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:58.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:57.991+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:59.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:58.932+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:59.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:58.932+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:59.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:58.932+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:28:59.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:58.933+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:59.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:58.838+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:59.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:58.838+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:59.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:58.838+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:28:59.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:58.839+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:28:59.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:58.972+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:59.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:58.972+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:59.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:58.972+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:28:59.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:58.973+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:00.026 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:28:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:28:59.749+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 23640 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:29:00.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:59.900+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:00.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:59.900+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:00.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:59.900+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:00.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:28:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:28:59.901+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:00.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:59.990+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:00.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:59.990+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:00.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:59.990+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:00.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:28:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:28:59.990+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:00.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:59.790+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:00.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:59.790+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:00.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:59.790+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:00.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:28:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:28:59.791+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:01.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:00.917+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:01.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:00.917+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:01.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:00.917+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:01.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:00.918+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:01.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:01.010+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:01.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:01.010+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:01.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:01.010+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:01.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:01.011+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:01.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:00.800+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:01.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:00.800+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:01.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:00.800+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:01.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:00.801+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:01.990 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:01.915+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:01.991 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:01.915+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:01.991 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:01.915+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:01.991 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:01.916+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:01.991 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:01.767+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:01.991 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:01.768+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:01.991 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:01.768+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:01.991 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:01.768+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:02.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:01.989+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:02.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:01.989+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:02.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:01.989+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:02.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:01.990+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:02.987 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:02.887+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:02.988 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:02.887+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:02.988 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:02.887+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:02.988 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:02.887+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:02.988 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:02.732+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:02.988 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:02.732+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:02.988 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:02.732+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:02.989 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:02.732+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:03.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:02.986+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:03.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:02.986+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:03.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:02.986+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:03.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:02.987+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:03.915 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:03.757+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:03.915 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:03.757+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:03.915 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:03.757+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:03.915 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:03.758+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:04.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:03.969+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:04.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:03.969+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:04.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:03.969+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:04.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:03.970+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:04.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:03.914+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:04.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:03.914+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:04.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:03.914+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:04.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:03.915+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:05.026 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:29:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:29:04.752+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 23722 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:29:05.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:04.918+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:05.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:04.918+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:05.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:04.918+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:05.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:04.918+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:05.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:04.796+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:05.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:04.796+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:05.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:04.796+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:05.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:04.797+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:05.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:04.959+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:05.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:04.959+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:05.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:04.959+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:05.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:04.960+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:06.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:05.919+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:06.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:05.919+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:06.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:05.919+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:06.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:05.920+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:06.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:06.000+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:06.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:06.000+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:06.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:06.000+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:06.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:06.002+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:06.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:05.830+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:06.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:05.830+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:06.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:05.830+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:06.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:05.831+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:07.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:06.915+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:07.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:06.915+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:07.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:06.915+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:07.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:06.916+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:07.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:06.854+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:07.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:06.854+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:07.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:06.854+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:07.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:06.855+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:07.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:06.969+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:07.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:06.969+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:07.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:06.969+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:07.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:06.970+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:08.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:07.963+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:08.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:07.963+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:08.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:07.963+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:08.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:07.964+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:08.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:07.857+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:08.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:07.857+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:08.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:07.857+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:08.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:07.858+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:08.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:07.970+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:08.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:07.970+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:08.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:07.970+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:08.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:07.971+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:09.178 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:08.862+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:09.178 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:08.862+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:09.178 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:08.862+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:09.179 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:08.863+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:09.179 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:08.982+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:09.179 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:08.982+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:09.179 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:08.982+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:09.179 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:08.982+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:09.180 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:09.004+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:09.180 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:09.004+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:09.180 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:09.004+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:09.180 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:09.005+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:10.026 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:29:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:29:09.755+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 23805 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:29:10.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:09.999+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:10.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:09.999+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:10.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:09.999+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:10.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:10.000+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:10.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:09.894+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:10.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:09.894+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:10.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:09.894+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:10.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:09.896+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:10.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:09.943+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:10.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:09.943+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:10.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:09.943+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:10.029 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:09.944+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:11.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:10.964+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:11.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:10.964+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:11.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:10.964+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:11.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:10.964+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:11.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:10.859+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:11.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:10.859+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:11.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:10.859+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:11.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:10.860+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:11.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:10.926+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:11.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:10.926+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:11.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:10.926+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:11.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:10.927+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:12.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:11.968+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:12.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:11.968+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:12.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:11.968+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:12.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:11.969+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:12.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:11.949+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:12.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:11.949+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:12.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:11.949+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:12.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:11.950+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:12.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:11.819+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:12.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:11.819+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:12.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:11.819+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:12.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:11.820+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:13.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:12.849+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:13.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:12.849+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:13.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:12.849+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:13.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:12.850+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:13.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:12.983+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:13.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:12.983+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:13.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:12.983+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:13.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:12.984+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:13.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:13.005+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:13.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:13.005+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:13.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:13.005+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:13.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:13.006+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:14.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:13.976+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:14.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:13.976+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:14.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:13.976+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:14.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:13.977+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:14.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:14.011+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:14.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:14.011+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:14.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:14.011+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:14.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:14.011+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:14.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:13.893+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:14.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:13.893+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:14.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:13.893+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:14.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:13.894+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:15.019 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:29:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:29:14.757+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 23887 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:29:15.020 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:14.863+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:15.020 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:14.863+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:15.020 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:14.863+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:15.020 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:14.864+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:15.021 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:14.996+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:15.021 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:14.996+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:15.021 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:14.996+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:15.021 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:14.997+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:15.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:15.019+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:15.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:15.019+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:15.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:15.019+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:15.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:15.020+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:16.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:16.058+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:16.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:16.058+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:16.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:16.058+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:16.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:16.058+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:16.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:15.993+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:16.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:15.993+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:16.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:15.993+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:16.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:15.994+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:16.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:15.885+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:16.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:15.885+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:16.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:15.885+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:16.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:15.886+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:17.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:16.975+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:17.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:16.975+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:17.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:16.975+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:17.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:16.976+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:17.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:17.018+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:17.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:17.018+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:17.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:17.018+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:17.279 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:17.019+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:17.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:16.896+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:17.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:16.896+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:17.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:16.896+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:17.280 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:16.897+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:18.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:17.910+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:18.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:17.910+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:18.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:17.910+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:18.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:17.911+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:18.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:17.972+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:18.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:17.972+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:18.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:17.972+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:18.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:17.973+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:18.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:17.933+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:18.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:17.933+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:18.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:17.933+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:18.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:17.934+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:19.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:18.946+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:19.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:18.946+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:19.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:18.946+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:19.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:18.947+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:19.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:18.926+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:19.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:18.926+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:19.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:18.926+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:19.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:18.926+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:19.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:18.939+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:19.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:18.939+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:19.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:18.939+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:19.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:18.940+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:20.026 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:29:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:29:19.760+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 23970 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:29:20.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:19.922+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:20.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:19.922+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:20.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:19.922+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:20.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:19.922+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:20.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:19.908+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:20.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:19.908+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:20.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:19.908+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:20.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:19.909+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:20.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:19.969+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:20.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:19.969+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:20.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:19.969+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:20.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:19.970+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:21.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:20.940+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:21.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:20.940+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:21.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:20.940+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:21.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:20.941+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:21.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:20.937+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:21.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:20.937+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:21.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:20.937+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:21.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:20.938+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:21.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:20.916+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:21.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:20.916+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:21.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:20.916+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:21.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:20.917+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:21.983 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:21.911+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:21.984 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:21.911+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:21.984 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:21.911+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:21.984 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:21.911+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:21.984 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:21.879+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:21.985 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:21.879+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:21.985 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:21.879+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:21.985 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:21.879+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:22.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:21.982+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:22.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:21.982+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:22.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:21.982+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:22.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:21.983+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:23.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:22.925+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:23.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:22.925+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:23.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:22.925+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:23.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:22.926+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:23.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:22.857+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:23.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:22.857+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:23.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:22.857+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:23.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:22.857+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:23.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:22.935+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:23.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:22.935+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:23.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:22.935+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:23.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:22.936+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:24.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:23.885+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:24.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:23.885+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:24.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:23.885+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:24.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:23.885+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:24.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:23.965+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:24.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:23.965+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:24.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:23.965+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:24.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:23.966+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:24.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:23.982+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:24.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:23.982+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:24.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:23.982+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:24.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:23.983+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:25.026 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:29:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:29:24.763+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 24052 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:29:25.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:24.992+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:25.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:24.992+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:25.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:24.992+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:25.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:24.993+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:25.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:24.843+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:25.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:24.843+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:25.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:24.843+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:25.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:24.843+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:25.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:24.936+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:25.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:24.936+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:25.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:24.936+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:25.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:24.937+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:26.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:25.881+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:26.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:25.881+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:26.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:25.881+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:26.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:25.882+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:26.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:25.976+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:26.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:25.976+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:26.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:25.976+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:26.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:25.977+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:26.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:25.919+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:26.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:25.919+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:26.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:25.919+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:26.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:25.920+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:27.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:26.942+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:27.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:26.942+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:27.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:26.942+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:27.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:26.943+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:27.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:26.885+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:27.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:26.885+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:27.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:26.885+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:27.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:26.886+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:27.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:27.022+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:27.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:27.022+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:27.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:27.022+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:27.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:27.023+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:28.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:27.969+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:28.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:27.969+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:28.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:27.969+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:28.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:27.970+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:28.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:27.858+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:28.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:27.858+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:28.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:27.858+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:28.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:27.859+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:28.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:28.052+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:28.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:28.052+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:28.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:28.052+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:28.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:28.053+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:29.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:29.012+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:29.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:29.012+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:29.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:29.012+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:29.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:29.012+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:29.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:29.025+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:29.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:29.025+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:29.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:29.025+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:29.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:29.026+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:29.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:28.898+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:29.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:28.898+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:29.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:28.898+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:29.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:28.899+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:30.026 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:29:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:29:29.765+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 24134 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:29:30.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:29.942+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:30.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:29.943+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:30.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:29.943+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:30.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:29.943+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:30.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:30.009+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:30.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:30.009+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:30.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:30.009+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:30.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:30.010+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:30.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:30.052+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:30.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:30.053+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:30.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:30.053+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:30.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:30.053+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:31.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:31.025+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:31.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:31.025+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:31.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:31.025+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:31.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:31.026+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:31.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:30.992+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:31.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:30.993+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:31.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:30.993+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:31.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:30.993+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:31.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:31.008+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:31.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:31.008+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:31.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:31.008+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:31.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:31.009+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:32.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:31.993+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:32.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:31.993+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:32.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:31.993+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:32.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:31.994+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:32.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:32.017+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:32.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:32.017+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:32.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:32.017+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:32.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:32.018+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:32.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:32.018+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:32.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:32.018+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:32.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:32.018+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:32.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:32.019+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:33.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:33.023+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:33.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:33.023+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:33.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:33.023+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:33.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:33.024+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:33.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:33.031+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:33.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:33.031+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:33.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:33.031+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:33.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:33.032+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:33.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:33.034+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:33.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:33.034+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:33.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:33.034+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:33.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:33.035+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:34.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:34.056+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:34.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:34.056+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:34.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:34.056+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:34.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:34.057+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:34.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:34.044+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:34.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:34.044+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:34.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:34.044+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:34.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:34.044+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:34.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:34.008+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:34.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:34.008+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:34.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:34.008+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:34.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:34.009+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:35.026 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:29:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:29:34.768+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 24216 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:29:35.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:34.995+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:35.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:34.995+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:35.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:34.995+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:35.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:34.996+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:35.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:35.055+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:35.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:35.055+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:35.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:35.055+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:35.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:35.056+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:35.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:35.064+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:35.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:35.064+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:35.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:35.064+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:35.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:35.065+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:36.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:36.068+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:36.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:36.068+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:36.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:36.068+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:36.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:36.069+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:36.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:35.980+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:36.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:35.980+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:36.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:35.980+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:36.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:35.981+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:36.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:36.087+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:36.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:36.087+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:36.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:36.087+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:36.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:36.089+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:37.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:37.022+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:37.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:37.022+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:37.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:37.022+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:37.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:37.023+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:37.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:36.961+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:37.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:36.961+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:37.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:36.961+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:37.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:36.962+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:37.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:37.052+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:37.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:37.052+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:37.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:37.052+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:37.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:37.052+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:38.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:37.953+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:38.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:37.953+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:38.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:37.953+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:38.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:37.954+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:38.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:37.979+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:38.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:37.979+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:38.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:37.979+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:38.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:37.979+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:38.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:38.013+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:38.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:38.013+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:38.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:38.013+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:38.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:38.014+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:39.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:38.940+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:39.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:38.940+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:39.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:38.940+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:39.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:38.941+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:39.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:38.998+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:39.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:38.998+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:39.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:38.998+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:39.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:38.999+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:39.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:39.030+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:39.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:39.030+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:39.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:39.030+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:39.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:39.031+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:40.026 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:29:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:29:39.771+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 24299 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:29:40.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:39.990+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:40.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:39.990+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:40.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:39.990+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:40.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:39.991+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:40.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:40.011+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:40.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:40.011+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:40.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:40.011+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:40.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:40.011+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:40.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:40.042+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:40.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:40.042+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:40.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:40.042+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:40.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:40.043+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:41.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:40.975+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:41.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:40.975+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:41.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:40.975+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:41.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:40.975+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:41.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:41.016+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:41.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:41.016+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:41.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:41.016+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:41.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:41.017+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:41.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:41.015+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:41.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:41.015+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:41.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:41.015+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:41.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:41.016+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:42.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:42.014+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:42.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:42.014+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:42.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:42.014+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:42.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:42.015+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:42.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:42.051+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:42.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:42.051+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:42.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:42.051+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:42.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:42.052+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:42.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:42.021+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:42.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:42.021+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:42.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:42.021+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:42.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:42.023+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:43.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:43.025+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:43.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:43.025+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:43.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:43.025+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:43.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:43.026+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:43.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:43.014+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:43.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:43.014+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:43.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:43.014+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:43.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:43.015+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:43.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:43.047+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:43.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:43.047+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:43.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:43.047+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:43.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:43.048+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:44.355 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:44.063+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:44.355 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:44.063+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:44.355 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:44.063+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:44.355 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:44.064+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:44.355 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:44.065+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:44.356 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:44.065+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:44.356 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:44.065+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:44.356 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:44.066+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:44.356 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:44.061+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:44.356 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:44.061+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:44.356 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:44.061+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:44.356 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:44.061+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:45.026 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:29:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:29:44.773+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 24381 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:29:45.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:45.016+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:45.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:45.016+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:45.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:45.016+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:45.028 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:45.017+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:45.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:45.033+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:45.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:45.033+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:45.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:45.033+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:45.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:45.034+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:45.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:45.108+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:45.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:45.108+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:45.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:45.108+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:45.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:45.108+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:46.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:46.155+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:46.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:46.155+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:46.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:46.155+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:46.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:46.155+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:46.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:45.999+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:46.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:45.999+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:46.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:45.999+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:46.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:46.000+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:46.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:46.004+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:46.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:46.004+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:46.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:46.004+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:46.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:46.004+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:47.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:47.157+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:47.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:47.157+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:47.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:47.157+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:47.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:47.157+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:47.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:46.959+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:47.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:46.959+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:47.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:46.959+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:47.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:46.960+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:47.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:47.044+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:47.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:47.044+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:47.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:47.044+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:47.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:47.045+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:48.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:48.120+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:48.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:48.120+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:48.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:48.120+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:48.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:48.121+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:48.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:47.922+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:48.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:47.922+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:48.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:47.922+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:48.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:47.922+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:48.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:48.069+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:48.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:48.069+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:48.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:48.069+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:48.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:48.071+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:49.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:49.091+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:49.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:49.091+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:49.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:49.091+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:49.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:49.092+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:49.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:49.147+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:49.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:49.147+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:49.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:49.147+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:49.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:49.147+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:49.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:48.913+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:49.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:48.913+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:49.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:48.913+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:49.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:48.914+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:50.045 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:29:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:29:49.776+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 24464 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:29:50.045 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:49.868+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:50.045 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:49.868+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:50.045 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:49.868+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:50.045 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:49.869+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:50.500 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:50.131+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:50.500 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:50.131+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:50.500 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:50.131+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:50.500 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:50.132+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:50.501 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:50.044+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:50.501 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:50.044+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:50.501 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:50.044+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:50.501 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:50.044+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:51.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:51.038+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:51.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:51.038+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:51.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:51.038+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:51.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:51.038+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:51.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:51.111+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:51.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:51.111+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:51.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:51.111+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:51.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:51.112+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:51.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:50.875+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:51.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:50.875+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:51.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:50.875+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:51.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:50.876+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:52.026 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:51.836+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:52.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:51.836+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:52.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:51.836+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:52.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:51.838+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:52.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:52.081+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:52.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:52.081+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:52.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:52.081+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:52.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:52.081+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:52.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:52.062+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:52.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:52.062+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:52.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:52.062+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:52.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:52.063+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:53.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:52.836+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:53.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:52.836+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:53.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:52.836+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:53.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:52.836+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:53.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:53.058+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:53.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:53.058+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:53.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:53.058+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:53.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:53.059+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:53.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:53.025+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:53.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:53.025+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:53.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:53.025+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:53.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:53.025+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:54.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:53.858+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:54.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:53.858+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:54.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:53.858+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:54.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:53.859+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:54.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:54.035+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:54.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:54.035+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:54.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:54.035+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:54.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:54.036+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:54.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:54.069+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:54.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:54.069+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:54.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:54.069+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:54.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:54.070+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:55.036 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:29:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:29:54.779+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 24546 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:29:55.037 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:54.857+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:55.037 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:54.857+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:55.037 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:54.857+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:55.037 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:54.857+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:55.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:55.036+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:55.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:55.036+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:55.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:55.036+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:55.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:55.037+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:55.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:55.036+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:55.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:55.036+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:55.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:55.036+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:55.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:55.037+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:56.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:55.892+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:56.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:55.893+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:56.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:55.893+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:56.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:55.894+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:56.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:56.055+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:56.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:56.055+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:56.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:56.055+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:56.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:56.056+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:56.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:56.083+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:56.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:56.083+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:56.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:56.083+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:56.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:56.084+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:57.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:56.906+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:57.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:56.906+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:57.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:56.906+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:57.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:56.906+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:57.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:57.013+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:57.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:57.013+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:57.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:57.013+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:57.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:57.014+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:57.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:57.127+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:57.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:57.127+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:57.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:57.127+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:57.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:57.128+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:58.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:57.929+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:58.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:57.929+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:58.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:57.929+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:58.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:57.930+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:58.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:58.120+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:58.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:58.120+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:58.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:58.120+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:58.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:58.121+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:58.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:58.025+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:58.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:58.025+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:58.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:58.025+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:58.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:58.026+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:59.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:58.885+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:59.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:58.885+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:59.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:58.885+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:29:59.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:58.886+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:59.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:59.073+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:59.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:59.073+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:59.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:59.073+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:29:59.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:29:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:29:59.074+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:29:59.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:59.092+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:59.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:59.092+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:59.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:59.092+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:29:59.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:29:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:29:59.093+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:00.091 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:29:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:29:59.781+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 24629 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:30:00.092 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:59.922+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:00.092 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:59.922+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:00.092 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:59.922+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:00.092 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:29:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:29:59.923+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:00.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:00.090+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:00.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:00.090+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:00.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:00.090+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:00.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:00.092+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:00.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:00.124+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:00.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:00.124+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:00.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:00.124+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:00.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:00.124+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:01.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:01.062+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:01.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:01.062+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:01.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:01.062+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:01.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:01.063+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:01.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:01.085+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:01.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:01.085+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:01.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:01.085+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:01.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:01.086+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:01.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:00.880+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:01.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:00.880+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:01.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:00.880+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:01.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:00.881+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:02.026 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:01.914+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:02.026 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:01.914+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:02.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:01.914+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:02.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:01.916+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:02.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:02.073+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:02.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:02.073+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:02.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:02.073+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:02.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:02.074+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:02.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:02.127+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:02.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:02.127+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:02.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:02.127+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:02.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:02.128+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:03.164 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:02.897+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:03.165 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:02.897+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:03.165 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:02.897+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:03.165 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:02.898+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:03.165 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:03.035+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:03.166 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:03.035+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:03.166 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:03.035+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:03.166 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:03.035+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:03.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:03.164+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:03.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:03.164+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:03.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:03.164+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:03.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:03.165+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:04.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:04.064+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:04.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:04.064+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:04.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:04.064+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:04.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:04.065+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:04.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:04.166+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:04.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:04.166+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:04.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:04.166+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:04.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:04.167+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:04.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:03.926+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:04.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:03.926+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:04.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:03.926+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:04.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:03.927+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:05.198 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:30:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:30:04.784+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 24711 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:30:05.198 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:04.963+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:05.198 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:04.963+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:05.199 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:04.963+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:05.199 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:04.964+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:05.199 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:05.029+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:05.199 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:05.029+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:05.199 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:05.029+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:05.199 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:05.030+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:05.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:05.197+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:05.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:05.197+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:05.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:05.197+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:05.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:05.198+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:06.188 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:06.010+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:06.188 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:06.010+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:06.188 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:06.010+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:06.189 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:06.011+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:06.189 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:06.044+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:06.189 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:06.044+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:06.189 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:06.044+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:06.189 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:06.045+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:06.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:06.187+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:06.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:06.187+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:06.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:06.187+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:06.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:06.188+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:07.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:07.032+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:07.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:07.032+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:07.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:07.032+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:07.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:07.033+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:07.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:07.040+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:07.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:07.040+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:07.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:07.040+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:07.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:07.041+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:07.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:07.154+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:07.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:07.154+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:07.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:07.154+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:07.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:07.155+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:08.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:08.183+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:08.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:08.183+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:08.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:08.183+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:08.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:08.184+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:08.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:08.008+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:08.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:08.008+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:08.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:08.008+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:08.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:08.009+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:08.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:08.086+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:08.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:08.086+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:08.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:08.086+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:08.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:08.087+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:09.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:09.002+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:09.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:09.002+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:09.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:09.002+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:09.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:09.003+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:09.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:09.074+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:09.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:09.074+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:09.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:09.074+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:09.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:09.075+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:09.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:09.145+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:09.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:09.145+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:09.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:09.145+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:09.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:09.145+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:10.117 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:30:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:30:09.787+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 24794 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:30:10.117 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:10.014+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:10.117 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:10.014+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:10.117 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:10.014+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:10.117 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:10.015+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:10.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:10.194+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:10.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:10.194+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:10.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:10.194+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:10.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:10.194+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:10.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:10.116+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:10.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:10.116+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:10.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:10.116+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:10.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:10.117+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:11.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:11.149+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:11.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:11.149+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:11.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:11.149+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:11.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:11.150+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:11.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:11.012+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:11.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:11.012+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:11.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:11.012+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:11.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:11.013+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:11.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:11.227+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:11.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:11.227+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:11.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:11.227+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:11.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:11.227+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:12.271 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:12.030+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:12.272 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:12.030+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:12.272 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:12.030+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:12.272 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:12.031+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:12.272 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:12.168+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:12.272 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:12.168+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:12.272 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:12.168+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:12.273 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:12.169+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:12.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:12.271+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:12.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:12.271+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:12.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:12.271+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:12.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:12.272+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:13.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:13.260+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:13.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:13.260+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:13.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:13.260+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:13.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:13.261+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:13.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:13.163+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:13.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:13.163+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:13.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:13.163+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:13.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:13.164+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:13.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:13.069+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:13.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:13.069+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:13.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:13.069+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:13.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:13.070+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:14.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:14.217+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:14.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:14.217+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:14.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:14.217+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:14.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:14.217+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:14.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:14.188+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:14.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:14.188+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:14.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:14.188+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:14.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:14.190+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:14.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:14.053+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:14.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:14.053+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:14.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:14.053+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:14.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:14.054+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:15.063 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:30:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:30:14.790+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 24877 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:30:15.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:15.161+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:15.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:15.161+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:15.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:15.161+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:15.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:15.162+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:15.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:15.063+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:15.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:15.063+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:15.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:15.063+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:15.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:15.064+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:15.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:15.224+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:15.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:15.224+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:15.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:15.224+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:15.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:15.225+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:16.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:16.035+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:16.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:16.035+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:16.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:16.035+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:16.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:16.036+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:16.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:16.211+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:16.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:16.211+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:16.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:16.211+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:16.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:16.212+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:16.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:16.253+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:16.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:16.253+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:16.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:16.253+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:16.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:16.254+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:17.529 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:17.233+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:17.529 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:17.233+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:17.529 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:17.233+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:17.529 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:17.234+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:17.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:17.054+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:17.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:17.054+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:17.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:17.054+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:17.530 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:17.055+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:17.530 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:17.276+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:17.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:17.276+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:17.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:17.276+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:17.531 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:17.277+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:18.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:18.237+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:18.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:18.237+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:18.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:18.237+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:18.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:18.238+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:18.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:18.044+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:18.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:18.044+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:18.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:18.044+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:18.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:18.045+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:18.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:18.288+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:18.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:18.288+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:18.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:18.288+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:18.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:18.289+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:19.384 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:19.264+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:19.384 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:19.264+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:19.384 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:19.264+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:19.384 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:19.265+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:19.385 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:19.276+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:19.385 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:19.276+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:19.385 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:19.276+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:19.385 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:19.277+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:19.385 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:19.048+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:19.386 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:19.048+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:19.386 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:19.048+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:19.386 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:19.048+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:20.164 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:30:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:30:19.793+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 24960 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:30:20.164 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:20.007+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:20.164 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:20.007+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:20.165 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:20.007+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:20.165 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:20.008+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:20.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:20.250+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:20.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:20.250+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:20.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:20.250+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:20.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:20.250+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:20.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:20.217+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:20.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:20.217+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:20.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:20.217+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:20.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:20.218+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:21.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:21.234+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:21.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:21.234+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:21.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:21.234+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:21.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:21.235+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:21.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:21.023+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:21.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:21.023+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:21.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:21.023+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:21.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:21.023+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:21.734 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:21.294+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:21.734 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:21.294+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:21.734 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:21.294+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:21.734 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:21.294+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:22.242 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:21.985+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:22.242 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:21.985+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:22.242 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:21.985+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:22.242 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:21.985+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:22.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:22.333+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:22.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:22.333+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:22.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:22.333+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:22.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:22.333+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:22.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:22.241+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:22.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:22.241+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:22.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:22.241+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:22.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:22.243+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:23.248 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:22.950+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:23.249 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:22.950+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:23.249 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:22.950+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:23.249 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:22.951+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:23.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:23.247+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:23.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:23.248+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:23.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:23.248+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:23.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:23.248+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:23.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:23.319+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:23.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:23.319+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:23.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:23.319+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:23.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:23.320+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:24.271 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:23.950+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:24.271 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:23.950+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:24.271 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:23.950+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:24.271 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:23.950+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:24.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:24.291+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:24.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:24.291+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:24.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:24.291+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:24.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:24.292+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:24.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:24.270+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:24.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:24.270+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:24.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:24.270+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:24.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:24.271+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:25.276 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:30:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:30:24.796+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 25043 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:30:25.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:24.916+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:25.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:24.916+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:25.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:24.916+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:25.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:24.916+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:25.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:25.313+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:25.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:25.313+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:25.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:25.313+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:25.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:25.314+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:25.778 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:25.293+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:25.778 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:25.293+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:25.778 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:25.293+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:25.779 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:25.294+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:26.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:25.936+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:26.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:25.936+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:26.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:25.936+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:26.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:25.937+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:26.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:26.313+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:26.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:26.313+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:26.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:26.313+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:26.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:26.314+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:26.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:26.286+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:26.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:26.287+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:26.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:26.287+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:26.778 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:26.288+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:27.274 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:26.971+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:27.274 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:26.971+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:27.274 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:26.971+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:27.274 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:26.972+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:27.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:27.286+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:27.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:27.286+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:27.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:27.286+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:27.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:27.287+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:27.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:27.273+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:27.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:27.273+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:27.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:27.273+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:27.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:27.274+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:28.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:27.948+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:28.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:27.948+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:28.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:27.948+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:28.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:27.949+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:28.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:28.286+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:28.807 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:28.286+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:28.807 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:28.286+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:28.807 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:28.287+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:28.808 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:28.287+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:28.808 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:28.287+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:28.808 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:28.287+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:28.808 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:28.288+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:29.254 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:28.983+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:29.254 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:28.983+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:29.254 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:28.983+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:29.254 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:28.984+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:29.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:29.328+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:29.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:29.328+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:29.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:29.328+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:29.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:29.329+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:29.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:29.253+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:29.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:29.253+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:29.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:29.253+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:29.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:29.254+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:30.165 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:30:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:30:29.799+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 25126 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:30:30.166 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:30.008+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:30.166 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:30.008+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:30.166 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:30.008+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:30.166 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:30.009+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:30.435 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:30.231+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:30.435 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:30.231+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:30.435 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:30.231+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:30.435 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:30.232+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:30.435 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:30.322+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:30.435 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:30.322+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:30.436 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:30.322+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:30.436 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:30.323+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:31.287 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:31.245+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:31.287 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:31.245+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:31.287 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:31.245+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:31.287 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:31.245+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:31.287 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:31.035+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:31.288 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:31.035+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:31.288 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:31.035+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:31.288 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:31.036+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:31.734 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:31.286+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:31.735 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:31.286+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:31.735 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:31.286+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:31.735 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:31.287+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:32.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:32.005+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:32.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:32.005+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:32.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:32.005+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:32.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:32.006+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:32.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:32.242+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:32.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:32.242+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:32.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:32.242+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:32.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:32.242+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:32.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:32.324+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:32.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:32.324+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:32.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:32.324+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:32.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:32.326+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:33.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:33.025+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:33.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:33.025+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:33.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:33.025+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:33.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:33.026+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:33.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:33.244+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:33.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:33.244+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:33.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:33.244+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:33.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:33.245+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:33.620 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:33.326+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:33.621 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:33.326+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:33.621 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:33.326+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:33.621 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:33.326+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:34.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:34.204+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:34.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:34.204+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:34.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:34.204+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:34.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:34.205+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:34.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:33.976+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:34.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:33.976+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:34.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:33.976+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:34.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:33.977+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:34.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:34.359+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:34.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:34.359+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:34.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:34.359+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:34.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:34.360+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:35.180 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:30:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:30:34.802+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 25209 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:30:35.181 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:34.954+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:35.181 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:34.954+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:35.181 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:34.954+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:35.181 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:34.954+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:35.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:35.324+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:35.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:35.324+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:35.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:35.324+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:35.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:35.325+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:35.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:35.180+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:35.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:35.180+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:35.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:35.180+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:35.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:35.181+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:36.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:36.151+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:36.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:36.151+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:36.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:36.151+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:36.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:36.152+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:36.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:35.926+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:36.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:35.926+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:36.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:35.926+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:36.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:35.927+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:36.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:36.368+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:36.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:36.368+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:36.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:36.368+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:36.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:36.369+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:37.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:36.944+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:37.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:36.944+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:37.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:36.944+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:37.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:36.946+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:37.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:37.121+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:37.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:37.121+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:37.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:37.121+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:37.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:37.122+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:37.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:37.349+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:37.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:37.349+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:37.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:37.349+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:37.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:37.350+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:38.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:37.915+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:38.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:37.915+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:38.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:37.915+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:38.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:37.916+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:38.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:38.147+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:38.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:38.147+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:38.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:38.147+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:38.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:38.148+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:38.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:38.305+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:38.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:38.305+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:38.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:38.305+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:38.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:38.306+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:39.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:38.944+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:39.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:38.944+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:39.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:38.944+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:39.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:38.944+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:39.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:39.127+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:39.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:39.127+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:39.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:39.127+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:39.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:39.128+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:39.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:39.275+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:39.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:39.275+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:39.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:39.275+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:39.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:39.276+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:40.142 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:30:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:30:39.805+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 25290 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:30:40.142 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:39.925+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:40.143 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:39.925+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:40.143 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:39.925+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:40.143 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:39.925+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:40.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:40.141+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:40.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:40.141+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:40.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:40.141+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:40.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:40.142+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:40.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:40.279+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:40.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:40.279+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:40.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:40.279+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:40.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:40.280+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:41.254 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:40.914+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:41.254 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:40.914+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:41.255 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:40.914+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:41.255 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:40.914+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:41.255 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:41.148+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:41.255 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:41.148+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:41.255 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:41.148+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:41.255 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:41.148+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:41.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:41.253+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:41.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:41.253+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:41.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:41.253+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:41.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:41.254+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:42.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:41.920+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:42.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:41.920+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:42.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:41.920+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:42.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:41.921+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:42.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:42.121+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:42.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:42.121+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:42.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:42.121+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:42.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:42.122+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:42.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:42.242+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:42.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:42.242+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:42.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:42.242+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:42.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:42.243+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:43.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:42.966+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:43.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:42.966+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:43.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:42.966+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:43.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:42.967+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:43.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:43.121+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:43.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:43.121+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:43.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:43.121+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:43.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:43.122+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:43.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:43.239+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:43.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:43.239+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:43.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:43.239+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:43.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:43.240+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:44.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:44.159+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:44.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:44.159+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:44.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:44.159+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:44.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:44.160+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:44.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:44.204+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:44.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:44.204+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:44.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:44.204+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:44.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:44.205+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:44.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:44.011+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:44.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:44.011+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:44.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:44.011+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:44.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:44.011+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:45.129 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:30:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:30:44.808+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 25374 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:30:45.129 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:45.057+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:45.129 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:45.057+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:45.129 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:45.057+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:45.130 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:45.058+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:45.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:45.128+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:45.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:45.128+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:45.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:45.128+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:45.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:45.129+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:45.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:45.244+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:45.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:45.244+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:45.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:45.244+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:45.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:45.244+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:46.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:46.104+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:46.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:46.104+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:46.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:46.104+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:46.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:46.104+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:46.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:46.111+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:46.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:46.111+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:46.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:46.111+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:46.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:46.112+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:46.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:46.218+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:46.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:46.218+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:46.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:46.218+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:46.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:46.218+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:47.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:47.092+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:47.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:47.092+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:47.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:47.092+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:47.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:47.094+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:47.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:47.141+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:47.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:47.141+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:47.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:47.141+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:47.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:47.142+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:47.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:47.224+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:47.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:47.224+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:47.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:47.224+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:47.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:47.225+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:48.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:48.095+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:48.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:48.095+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:48.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:48.095+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:48.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:48.096+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:48.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:48.121+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:48.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:48.121+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:48.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:48.121+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:48.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:48.122+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:48.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:48.191+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:48.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:48.191+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:48.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:48.191+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:48.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:48.192+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:49.384 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:49.152+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:49.385 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:49.152+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:49.385 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:49.152+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:49.385 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:49.153+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:49.385 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:49.049+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:49.386 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:49.049+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:49.386 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:49.049+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:49.386 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:49.050+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:49.386 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:49.120+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:49.386 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:49.120+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:49.386 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:49.120+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:49.387 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:49.121+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:50.125 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:30:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:30:49.811+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 25456 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:30:50.125 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:50.052+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:50.126 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:50.052+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:50.126 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:50.052+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:50.126 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:50.053+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:50.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:50.124+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:50.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:50.124+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:50.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:50.124+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:50.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:50.125+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:50.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:50.139+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:50.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:50.139+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:50.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:50.139+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:50.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:50.140+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:51.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:51.080+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:51.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:51.080+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:51.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:51.080+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:51.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:51.080+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:51.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:51.132+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:51.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:51.132+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:51.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:51.132+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:51.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:51.133+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:51.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:51.108+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:51.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:51.108+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:51.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:51.108+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:51.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:51.108+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:52.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:52.094+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:52.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:52.094+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:52.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:52.094+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:52.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:52.095+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:52.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:52.085+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:52.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:52.085+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:52.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:52.085+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:52.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:52.085+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:52.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:52.078+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:52.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:52.078+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:52.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:52.078+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:52.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:52.079+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:53.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:53.056+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:53.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:53.056+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:53.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:53.056+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:53.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:53.057+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:53.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:53.037+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:53.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:53.037+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:53.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:53.037+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:53.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:53.038+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:53.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:53.116+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:53.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:53.116+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:53.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:53.116+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:53.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:53.118+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:54.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:54.106+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:54.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:54.106+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:54.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:54.106+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:54.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:54.107+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:54.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:54.154+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:54.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:54.154+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:54.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:54.154+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:54.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:54.155+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:54.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:54.005+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:54.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:54.005+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:54.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:54.005+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:54.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:54.005+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:55.099 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:30:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:30:54.813+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 25540 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:30:55.099 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:54.966+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:55.099 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:54.966+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:55.099 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:54.966+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:55.099 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:54.967+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:55.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:55.198+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:55.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:55.198+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:55.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:55.198+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:55.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:55.199+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:55.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:55.098+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:55.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:55.098+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:55.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:55.098+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:55.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:55.099+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:56.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:56.138+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:56.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:56.138+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:56.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:56.138+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:56.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:56.139+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:56.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:55.921+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:56.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:55.921+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:56.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:55.921+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:56.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:55.921+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:56.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:56.165+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:56.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:56.165+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:56.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:56.165+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:56.280 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:56.166+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:57.140 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:56.873+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:57.140 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:56.873+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:57.140 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:56.873+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:57.140 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:56.873+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:57.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:57.147+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:57.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:57.147+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:57.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:57.147+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:57.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:57.148+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:57.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:57.139+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:57.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:57.139+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:57.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:57.139+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:57.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:57.140+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:58.154 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:57.874+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:58.154 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:57.874+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:58.154 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:57.874+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:58.154 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:57.875+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:58.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:58.153+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:58.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:58.153+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:58.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:58.153+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:58.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:58.154+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:58.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:58.159+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:58.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:58.159+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:58.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:58.159+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:58.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:58.160+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:59.113 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:58.835+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:59.113 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:58.835+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:59.113 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:58.835+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:30:59.114 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:58.836+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:59.384 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:59.191+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:59.385 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:59.191+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:59.385 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:59.191+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:30:59.385 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:30:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:30:59.192+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:30:59.385 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:59.113+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:59.385 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:59.113+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:59.385 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:59.113+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:30:59.386 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:30:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:30:59.113+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:00.143 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:30:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:30:59.816+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 25621 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:31:00.144 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:59.871+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:00.144 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:59.871+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:00.145 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:59.871+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:00.145 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:30:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:30:59.871+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:00.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:00.155+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:00.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:00.155+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:00.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:00.155+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:00.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:00.156+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:00.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:00.143+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:00.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:00.143+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:00.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:00.143+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:00.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:00.144+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:01.146 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:01.122+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:01.147 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:01.122+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:01.147 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:01.122+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:01.147 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:01.123+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:01.147 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:00.888+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:01.147 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:00.888+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:01.148 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:00.888+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:01.148 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:00.889+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:01.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:01.146+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:01.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:01.146+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:01.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:01.146+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:01.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:01.147+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:02.026 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:01.884+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:02.026 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:01.884+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:02.026 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:01.884+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:02.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:01.885+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:02.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:02.085+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:02.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:02.085+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:02.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:02.085+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:02.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:02.085+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:02.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:02.154+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:02.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:02.154+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:02.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:02.154+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:02.529 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:02.155+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:03.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:03.123+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:03.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:03.123+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:03.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:03.123+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:03.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:03.124+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:03.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:03.120+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:03.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:03.120+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:03.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:03.120+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:03.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:03.121+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:03.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:02.902+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:03.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:02.902+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:03.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:02.902+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:03.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:02.903+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:04.226 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:04.117+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:04.227 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:04.117+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:04.227 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:04.117+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:04.227 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:04.118+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:04.227 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:03.951+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:04.227 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:03.951+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:04.228 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:03.951+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:04.228 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:03.952+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:04.228 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:04.165+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:04.228 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:04.165+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:04.228 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:04.165+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:04.228 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:04.165+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:05.117 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:31:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:31:04.819+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 25705 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:31:05.118 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:04.931+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:05.118 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:04.931+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:05.118 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:04.931+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:05.118 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:04.932+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:05.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:05.117+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:05.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:05.117+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:05.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:05.117+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:05.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:05.118+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:05.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:05.161+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:05.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:05.161+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:05.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:05.161+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:05.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:05.161+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:06.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:05.966+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:06.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:05.966+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:06.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:05.966+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:06.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:05.967+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:06.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:06.092+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:06.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:06.092+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:06.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:06.092+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:06.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:06.093+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:06.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:06.170+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:06.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:06.170+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:06.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:06.170+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:06.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:06.171+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:07.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:07.175+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:07.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:07.175+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:07.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:07.175+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:07.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:07.176+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:07.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:07.126+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:07.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:07.126+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:07.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:07.126+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:07.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:07.128+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:07.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:06.940+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:07.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:06.940+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:07.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:06.940+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:07.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:06.941+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:08.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:08.160+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:08.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:08.160+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:08.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:08.160+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:08.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:08.161+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:08.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:08.104+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:08.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:08.104+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:08.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:08.104+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:08.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:08.105+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:08.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:07.961+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:08.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:07.961+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:08.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:07.961+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:08.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:07.962+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:09.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:09.141+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:09.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:09.141+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:09.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:09.141+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:09.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:09.142+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:09.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:09.089+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:09.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:09.089+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:09.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:09.089+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:09.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:09.090+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:09.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:09.000+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:09.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:09.000+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:09.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:09.000+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:09.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:09.001+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:10.168 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:31:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:31:09.822+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 25785 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:31:10.169 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:09.997+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:10.169 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:09.997+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:10.169 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:09.997+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:10.169 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:09.997+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:10.169 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:10.041+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:10.170 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:10.041+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:10.170 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:10.041+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:10.170 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:10.042+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:10.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:10.170+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:10.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:10.170+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:10.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:10.170+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:10.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:10.171+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:11.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:11.045+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:11.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:11.045+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:11.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:11.045+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:11.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:11.046+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:11.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:11.063+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:11.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:11.063+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:11.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:11.063+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:11.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:11.064+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:11.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:11.124+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:11.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:11.124+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:11.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:11.124+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:11.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:11.125+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:12.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:12.056+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:12.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:12.056+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:12.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:12.056+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:12.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:12.057+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:12.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:12.022+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:12.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:12.022+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:12.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:12.022+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:12.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:12.023+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:12.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:12.119+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:12.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:12.119+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:12.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:12.119+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:12.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:12.119+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:13.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:12.992+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:13.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:12.992+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:13.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:12.992+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:13.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:12.993+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:13.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:13.086+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:13.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:13.086+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:13.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:13.086+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:13.279 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:13.087+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:13.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:13.112+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:13.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:13.112+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:13.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:13.112+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:13.279 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:13.113+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:14.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:13.998+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:14.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:13.998+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:14.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:13.998+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:14.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:13.999+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:14.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:14.102+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:14.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:14.102+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:14.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:14.102+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:14.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:14.103+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:14.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:14.064+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:14.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:14.064+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:14.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:14.064+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:14.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:14.065+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:14.850 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:31:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:31:14.825+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 25868 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:31:15.116 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:14.981+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:15.116 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:14.981+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:15.116 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:14.981+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:15.116 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:14.982+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:15.116 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:15.017+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:15.117 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:15.017+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:15.117 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:15.017+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:15.117 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:15.018+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:15.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:15.115+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:15.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:15.115+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:15.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:15.115+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:15.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:15.116+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:16.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:16.109+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:16.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:16.109+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:16.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:16.109+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:16.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:16.109+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:16.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:15.993+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:16.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:15.993+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:16.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:15.993+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:16.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:15.994+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:16.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:16.008+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:16.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:16.008+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:16.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:16.008+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:16.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:16.008+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:17.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:17.043+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:17.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:17.043+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:17.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:17.043+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:17.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:17.044+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:17.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:17.101+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:17.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:17.101+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:17.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:17.101+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:17.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:17.101+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:17.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:17.038+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:17.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:17.038+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:17.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:17.038+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:17.528 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:17.038+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:18.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:18.060+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:18.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:18.060+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:18.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:18.060+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:18.527 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:18.061+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:18.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:18.071+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:18.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:18.071+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:18.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:18.071+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:18.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:18.072+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:18.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:18.098+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:18.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:18.098+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:18.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:18.098+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:18.528 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:18.098+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:19.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:19.052+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:19.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:19.052+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:19.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:19.052+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:19.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:19.053+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:19.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:19.061+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:19.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:19.061+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:19.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:19.061+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:19.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:19.061+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:19.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:19.024+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:19.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:19.024+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:19.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:19.024+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:19.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:19.025+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:20.169 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:31:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:31:19.828+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 25950 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:31:20.169 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:20.064+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:20.169 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:20.064+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:20.170 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:20.064+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:20.170 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:20.065+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:20.170 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:19.983+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:20.170 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:19.983+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:20.170 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:19.983+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:20.171 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:19.983+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:20.171 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:20.062+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:20.171 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:20.062+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:20.171 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:20.062+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:20.171 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:20.062+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:21.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:21.077+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:21.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:21.077+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:21.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:21.077+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:21.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:21.078+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:21.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:20.956+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:21.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:20.956+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:21.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:20.956+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:21.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:20.957+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:21.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:21.090+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:21.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:21.090+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:21.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:21.090+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:21.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:21.091+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:22.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:21.929+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:22.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:21.929+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:22.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:21.929+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:22.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:21.929+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:22.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:22.081+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:22.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:22.081+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:22.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:22.081+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:22.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:22.081+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:22.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:22.073+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:22.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:22.073+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:22.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:22.073+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:22.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:22.074+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:23.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:23.122+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:23.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:23.122+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:23.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:23.122+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:23.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:23.123+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:23.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:23.060+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:23.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:23.060+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:23.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:23.060+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:23.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:23.061+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:23.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:22.948+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:23.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:22.948+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:23.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:22.948+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:23.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:22.949+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:24.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:24.012+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:24.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:24.012+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:24.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:24.012+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:24.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:24.013+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:24.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:24.113+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:24.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:24.113+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:24.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:24.113+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:24.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:24.113+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:24.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:23.907+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:24.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:23.907+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:24.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:23.907+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:24.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:23.907+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:25.094 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:24.867+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:25.095 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:24.867+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:25.095 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:24.867+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:25.095 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:24.868+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:25.095 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:25.061+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:25.095 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:25.061+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:25.096 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:25.061+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:25.096 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:25.062+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:25.096 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:31:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:31:24.831+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 26034 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:31:25.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:25.094+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:25.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:25.094+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:25.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:25.094+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:25.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:25.095+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:26.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:25.887+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:26.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:25.887+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:26.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:25.887+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:26.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:25.888+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:26.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:26.082+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:26.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:26.082+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:26.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:26.082+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:26.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:26.083+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:26.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:26.085+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:26.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:26.085+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:26.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:26.085+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:26.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:26.086+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:27.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:27.066+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:27.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:27.066+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:27.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:27.066+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:27.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:27.068+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:27.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:26.854+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:27.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:26.854+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:27.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:26.854+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:27.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:26.855+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:27.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:27.059+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:27.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:27.060+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:27.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:27.060+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:27.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:27.060+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:28.075 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:27.814+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:28.076 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:27.814+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:28.076 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:27.814+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:28.076 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:27.814+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:28.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:28.076+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:28.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:28.076+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:28.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:28.076+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:28.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:28.077+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:28.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:28.075+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:28.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:28.075+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:28.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:28.075+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:28.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:28.076+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:29.045 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:28.776+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:29.046 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:28.776+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:29.046 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:28.776+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:29.046 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:28.777+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:29.384 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:29.045+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:29.384 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:29.045+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:29.384 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:29.045+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:29.385 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:29.046+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:29.385 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:29.069+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:29.385 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:29.069+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:29.385 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:29.069+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:29.385 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:29.069+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:30.070 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:31:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:31:29.834+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 26116 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:31:30.070 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:29.777+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:30.070 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:29.777+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:30.070 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:29.777+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:30.071 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:29.778+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:30.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:30.105+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:30.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:30.105+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:30.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:30.105+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:30.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:30.107+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:30.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:30.069+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:30.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:30.069+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:30.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:30.069+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:30.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:30.070+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:31.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:30.730+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:31.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:30.730+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:31.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:30.730+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:31.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:30.731+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:31.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:31.120+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:31.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:31.120+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:31.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:31.120+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:31.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:31.121+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:31.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:31.132+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:31.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:31.132+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:31.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:31.132+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:31.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:31.133+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:32.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:31.765+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:32.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:31.765+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:32.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:31.765+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:32.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:31.766+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:32.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:32.163+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:32.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:32.163+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:32.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:32.163+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:32.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:32.163+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:32.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:32.116+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:32.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:32.116+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:32.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:32.116+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:32.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:32.117+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:33.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:32.728+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:33.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:32.728+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:33.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:32.728+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:33.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:32.728+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:33.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:33.124+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:33.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:33.124+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:33.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:33.124+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:33.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:33.125+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:33.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:33.142+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:33.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:33.142+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:33.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:33.142+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:33.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:33.143+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:34.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:33.715+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:34.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:33.715+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:34.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:33.715+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:34.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:33.716+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:34.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:34.135+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:34.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:34.135+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:34.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:34.135+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:34.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:34.136+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:34.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:34.170+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:34.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:34.170+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:34.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:34.170+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:34.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:34.171+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:34.961 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:31:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:31:34.837+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 26198 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:31:34.962 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:34.709+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:34.962 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:34.709+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:34.962 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:34.709+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:34.962 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:34.710+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:35.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:35.140+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:35.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:35.140+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:35.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:35.140+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:35.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:35.141+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:35.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:35.096+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:35.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:35.096+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:35.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:35.096+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:35.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:35.097+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:36.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:35.730+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:36.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:35.730+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:36.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:35.730+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:36.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:35.731+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:36.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:36.066+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:36.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:36.066+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:36.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:36.066+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:36.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:36.066+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:36.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:36.175+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:36.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:36.175+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:36.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:36.175+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:36.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:36.176+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:37.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:36.712+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:37.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:36.712+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:37.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:36.712+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:37.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:36.712+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:37.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:37.065+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:37.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:37.065+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:37.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:37.065+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:37.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:37.066+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:37.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:37.159+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:37.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:37.159+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:37.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:37.159+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:37.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:37.160+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:38.020 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:37.743+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:38.020 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:37.743+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:38.020 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:37.743+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:38.020 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:37.743+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:38.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:38.202+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:38.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:38.202+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:38.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:38.202+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:38.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:38.203+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:38.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:38.020+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:38.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:38.020+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:38.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:38.020+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:38.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:38.021+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:39.047 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:38.778+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:39.047 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:38.778+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:39.047 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:38.778+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:39.047 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:38.778+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:39.384 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:39.046+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:39.384 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:39.046+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:39.385 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:39.046+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:39.385 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:39.047+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:39.385 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:39.239+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:39.385 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:39.239+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:39.385 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:39.239+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:39.385 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:39.239+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:40.026 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:31:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:31:39.840+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 26280 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:31:40.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:39.753+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:40.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:39.753+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:40.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:39.753+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:40.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:39.754+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:40.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:40.057+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:40.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:40.057+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:40.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:40.057+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:40.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:40.058+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:40.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:40.232+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:40.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:40.232+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:40.527 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:40.232+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:40.528 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:40.233+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:41.019 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:40.742+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:41.019 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:40.742+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:41.019 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:40.743+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:41.019 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:40.743+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:41.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:41.019+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:41.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:41.019+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:41.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:41.019+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:41.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:41.019+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:41.740 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:41.282+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:41.740 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:41.282+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:41.740 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:41.282+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:41.740 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:41.282+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:42.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:41.787+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:42.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:41.787+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:42.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:41.787+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:42.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:41.787+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:42.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:41.973+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:42.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:41.973+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:42.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:41.973+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:42.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:41.974+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:42.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:42.297+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:42.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:42.297+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:42.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:42.297+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:42.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:42.298+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:43.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:42.824+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:43.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:42.824+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:43.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:42.824+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:43.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:42.825+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:43.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:42.997+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:43.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:42.997+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:43.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:42.997+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:43.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:42.998+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:43.625 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:43.276+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:43.625 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:43.276+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:43.625 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:43.276+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:43.626 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:43.277+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:43.950 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:43.857+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:43.951 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:43.857+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:43.951 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:43.857+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:43.951 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:43.858+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:44.244 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:43.950+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:44.244 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:43.950+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:44.244 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:43.950+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:44.244 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:43.951+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:44.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:44.244+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:44.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:44.244+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:44.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:44.244+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:44.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:44.244+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:45.216 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:31:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:31:44.843+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 26363 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:31:45.216 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:44.831+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:45.216 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:44.831+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:45.216 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:44.831+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:45.217 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:44.831+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:45.217 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:44.921+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:45.217 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:44.921+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:45.217 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:44.921+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:45.217 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:44.922+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:45.525 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:45.216+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:45.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:45.216+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:45.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:45.216+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:45.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:45.217+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:46.183 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:45.784+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:46.184 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:45.784+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:46.184 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:45.784+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:46.184 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:45.785+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:46.184 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:45.919+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:46.184 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:45.919+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:46.184 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:45.919+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:46.185 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:45.920+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:46.525 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:46.183+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:46.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:46.183+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:46.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:46.183+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:46.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:46.184+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:47.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:46.758+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:47.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:46.758+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:47.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:46.758+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:47.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:46.759+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:47.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:46.949+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:47.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:46.949+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:47.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:46.949+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:47.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:46.950+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:47.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:47.179+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:47.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:47.179+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:47.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:47.179+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:47.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:47.180+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:48.026 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:47.964+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:48.026 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:47.964+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:48.026 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:47.964+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:48.026 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:47.965+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:48.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:47.730+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:48.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:47.730+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:48.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:47.730+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:48.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:47.731+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:48.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:48.156+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:48.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:48.156+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:48.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:48.156+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:48.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:48.157+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:49.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:48.707+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:49.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:48.707+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:49.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:48.707+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:49.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:48.708+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:49.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:48.916+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:49.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:48.916+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:49.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:48.916+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:49.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:48.917+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:49.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:49.182+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:49.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:49.182+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:49.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:49.182+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:49.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:49.183+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:49.971 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:31:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:31:49.846+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 26444 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:31:49.971 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:49.712+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:49.972 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:49.712+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:49.972 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:49.712+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:49.972 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:49.713+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:49.972 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:49.886+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:49.972 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:49.886+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:49.972 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:49.886+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:49.973 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:49.887+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:50.525 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:50.232+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:50.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:50.232+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:50.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:50.232+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:50.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:50.233+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:51.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:50.756+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:51.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:50.756+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:51.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:50.756+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:51.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:50.757+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:51.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:50.869+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:51.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:50.869+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:51.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:50.869+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:51.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:50.869+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:51.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:51.243+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:51.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:51.243+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:51.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:51.243+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:51.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:51.243+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:52.026 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:51.895+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:52.026 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:51.895+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:52.026 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:51.895+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:52.026 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:51.895+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:52.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:51.714+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:52.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:51.714+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:52.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:51.714+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:52.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:51.715+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:52.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:52.266+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:52.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:52.266+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:52.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:52.266+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:52.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:52.267+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:53.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:52.723+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:53.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:52.723+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:53.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:52.723+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:53.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:52.724+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:53.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:52.883+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:53.028 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:52.883+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:53.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:52.883+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:53.029 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:52.884+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:53.625 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:53.283+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:53.625 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:53.283+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:53.625 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:53.283+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:53.625 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:53.284+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:53.911 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:53.700+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:53.911 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:53.700+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:53.911 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:53.700+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:53.911 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:53.702+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:54.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:53.911+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:54.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:53.911+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:54.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:53.911+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:54.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:53.912+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:54.695 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:54.286+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:54.696 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:54.286+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:54.696 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:54.286+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:54.696 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:54.287+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:54.972 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:31:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:31:54.849+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 26526 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:31:54.973 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:54.696+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:54.973 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:54.696+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:54.973 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:54.696+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:54.973 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:54.697+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:54.973 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:54.936+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:54.974 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:54.936+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:54.974 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:54.936+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:54.974 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:54.937+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:55.525 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:55.240+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:55.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:55.240+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:55.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:55.240+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:55.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:55.241+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:56.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:55.684+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:56.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:55.684+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:56.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:55.684+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:56.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:55.684+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:56.026 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:55.919+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:56.026 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:55.919+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:56.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:55.919+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:56.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:55.919+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:56.670 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:56.287+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:56.671 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:56.287+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:56.671 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:56.287+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:56.671 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:56.287+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:56.957 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:56.670+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:56.958 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:56.670+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:56.958 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:56.670+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:56.958 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:56.671+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:57.269 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:56.958+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:57.270 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:56.958+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:57.270 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:56.958+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:57.270 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:56.959+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:57.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:57.270+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:57.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:57.270+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:57.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:57.270+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:57.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:57.271+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:58.001 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:57.677+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:58.002 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:57.677+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:58.002 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:57.677+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:58.002 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:57.678+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:58.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:58.002+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:58.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:58.002+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:58.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:58.002+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:58.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:58.002+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:58.686 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:58.305+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:58.686 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:58.305+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:58.686 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:58.305+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:58.686 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:58.306+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:58.987 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:58.686+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:58.987 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:58.686+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:58.988 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:58.686+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:58.988 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:58.687+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:59.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:58.987+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:59.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:58.987+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:59.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:58.987+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:31:59.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:58.988+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:59.708 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:59.311+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:59.709 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:59.311+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:59.709 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:59.311+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:31:59.709 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:31:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:31:59.312+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:31:59.978 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:31:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:31:59.853+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 26606 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:31:59.981 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:59.708+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:59.981 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:59.708+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:59.981 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:59.708+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:31:59.981 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:31:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:31:59.709+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:00.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:59.981+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:00.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:59.981+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:00.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:59.981+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:00.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:31:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:31:59.982+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:00.664 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:00.340+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:00.664 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:00.340+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:00.664 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:00.340+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:00.664 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:00.341+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:01.012 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:00.664+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:01.012 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:00.664+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:01.013 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:00.664+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:01.013 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:00.665+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:01.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:01.012+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:01.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:01.012+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:01.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:01.012+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:01.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:01.013+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:01.652 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:01.313+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:01.652 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:01.313+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:01.652 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:01.313+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:01.652 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:01.314+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:01.975 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:01.652+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:01.975 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:01.652+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:01.975 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:01.652+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:01.975 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:01.652+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:02.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:01.975+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:02.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:01.975+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:02.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:01.975+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:02.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:01.976+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:02.594 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:02.317+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:02.594 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:02.317+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:02.594 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:02.317+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:02.594 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:02.318+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:02.946 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:02.617+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:02.947 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:02.617+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:02.947 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:02.617+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:02.947 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:02.618+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:03.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:02.947+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:03.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:02.947+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:03.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:02.947+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:03.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:02.948+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:03.625 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:03.291+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:03.625 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:03.291+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:03.625 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:03.291+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:03.626 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:03.292+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:03.948 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:03.649+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:03.948 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:03.649+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:03.948 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:03.649+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:03.949 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:03.650+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:04.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:03.948+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:04.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:03.948+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:04.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:03.948+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:04.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:03.949+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:04.694 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:04.337+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:04.694 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:04.337+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:04.694 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:04.337+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:04.695 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:04.338+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:04.978 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:32:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:32:04.856+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 26690 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:32:04.979 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:04.694+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:04.979 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:04.694+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:04.979 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:04.694+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:04.979 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:04.695+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:05.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:04.995+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:05.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:04.995+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:05.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:04.995+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:05.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:04.995+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:05.654 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:05.355+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:05.655 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:05.355+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:05.655 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:05.355+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:05.655 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:05.356+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:06.014 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:05.655+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:06.014 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:05.655+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:06.014 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:05.655+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:06.014 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:05.656+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:06.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:06.014+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:06.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:06.014+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:06.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:06.014+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:06.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:06.015+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:06.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:06.387+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:06.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:06.387+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:06.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:06.387+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:06.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:06.388+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:06.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:06.632+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:06.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:06.632+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:06.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:06.632+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:06.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:06.633+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:07.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:07.005+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:07.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:07.005+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:07.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:07.005+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:07.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:07.006+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:07.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:07.382+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:07.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:07.382+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:07.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:07.382+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:07.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:07.383+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:07.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:07.617+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:07.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:07.617+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:07.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:07.617+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:07.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:07.618+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:08.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:07.960+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:08.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:07.960+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:08.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:07.960+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:08.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:07.961+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:08.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:08.428+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:08.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:08.428+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:08.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:08.428+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:08.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:08.429+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:08.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:08.593+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:08.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:08.593+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:08.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:08.593+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:08.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:08.594+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:09.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:08.940+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:09.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:08.940+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:09.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:08.940+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:09.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:08.940+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:09.559 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:09.430+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:09.559 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:09.430+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:09.559 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:09.430+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:09.559 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:09.431+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:09.859 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:09.559+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:09.859 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:09.559+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:09.859 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:09.559+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:09.860 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:09.560+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:10.173 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:32:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:32:09.859+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 26772 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:32:10.173 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:09.936+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:10.173 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:09.936+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:10.173 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:09.936+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:10.173 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:09.936+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:10.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:10.573+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:10.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:10.573+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:10.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:10.573+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:10.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:10.574+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:10.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:10.452+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:10.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:10.452+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:10.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:10.452+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:10.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:10.452+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:11.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:10.950+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:11.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:10.950+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:11.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:10.950+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:11.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:10.951+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:11.741 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:11.482+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:11.742 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:11.482+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:11.742 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:11.482+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:11.742 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:11.483+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:11.742 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:11.549+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:11.742 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:11.549+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:11.742 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:11.549+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:11.743 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:11.550+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:12.025 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:11.906+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:12.026 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:11.906+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:12.026 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:11.906+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:12.026 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:11.907+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:12.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:12.501+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:12.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:12.501+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:12.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:12.501+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:12.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:12.502+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:12.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:12.466+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:12.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:12.466+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:12.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:12.466+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:12.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:12.467+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:13.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:12.903+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:13.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:12.903+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:13.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:12.903+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:13.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:12.904+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:13.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:13.536+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:13.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:13.536+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:13.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:13.536+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:13.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:13.536+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:13.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:13.490+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:13.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:13.490+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:13.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:13.490+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:13.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:13.490+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:14.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:13.868+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:14.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:13.868+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:14.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:13.868+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:14.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:13.869+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:14.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:14.508+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:14.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:14.508+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:14.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:14.508+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:14.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:14.509+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:14.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:14.572+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:14.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:14.572+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:14.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:14.572+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:14.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:14.572+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:15.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:32:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:32:14.862+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 26856 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:32:15.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:14.888+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:15.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:14.888+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:15.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:14.888+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:15.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:14.889+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:15.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:15.560+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:15.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:15.560+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:15.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:15.560+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:15.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:15.561+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:15.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:15.518+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:15.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:15.518+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:15.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:15.518+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:15.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:15.519+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:16.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:15.924+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:16.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:15.924+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:16.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:15.924+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:16.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:15.924+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:16.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:16.562+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:16.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:16.562+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:16.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:16.562+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:16.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:16.563+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:16.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:16.479+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:16.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:16.479+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:16.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:16.479+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:16.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:16.480+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:17.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:16.961+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:17.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:16.961+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:17.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:16.961+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:17.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:16.962+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:17.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:17.432+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:17.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:17.432+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:17.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:17.432+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:17.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:17.433+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:17.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:17.562+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:17.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:17.562+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:17.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:17.562+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:17.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:17.563+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:18.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:17.992+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:18.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:17.992+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:18.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:17.992+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:18.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:17.993+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:18.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:18.479+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:18.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:18.479+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:18.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:18.479+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:18.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:18.480+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:18.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:18.594+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:18.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:18.594+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:18.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:18.594+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:18.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:18.595+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:19.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:18.998+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:19.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:18.998+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:19.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:18.998+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:19.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:18.999+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:19.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:19.493+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:19.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:19.493+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:19.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:19.493+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:19.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:19.494+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:19.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:19.592+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:19.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:19.592+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:19.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:19.592+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:19.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:19.593+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:20.173 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:32:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:32:19.865+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 26938 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:32:20.173 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:19.979+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:20.174 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:19.980+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:20.174 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:19.980+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:20.174 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:19.980+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:20.997 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:20.538+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:20.997 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:20.538+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:20.998 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:20.538+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:20.998 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:20.539+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:20.998 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:20.628+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:20.998 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:20.628+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:20.998 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:20.628+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:20.998 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:20.629+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:21.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:20.997+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:21.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:20.997+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:21.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:20.997+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:21.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:20.999+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:21.994 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:21.545+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:21.994 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:21.545+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:21.994 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:21.545+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:21.994 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:21.546+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:21.995 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:21.604+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:21.995 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:21.604+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:21.995 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:21.604+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:21.995 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:21.604+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:22.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:21.994+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:22.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:21.994+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:22.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:21.994+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:22.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:21.995+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:23.007 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:22.641+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:23.007 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:22.641+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:23.007 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:22.641+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:23.007 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:22.642+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:23.007 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:22.555+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:23.008 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:22.555+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:23.008 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:22.555+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:23.008 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:22.556+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:23.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:23.007+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:23.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:23.007+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:23.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:23.007+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:23.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:23.008+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:24.011 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:23.690+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:24.011 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:23.690+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:24.012 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:23.690+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:24.012 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:23.690+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:24.012 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:23.530+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:24.012 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:23.530+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:24.012 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:23.530+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:24.012 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:23.530+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:24.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:24.011+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:24.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:24.011+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:24.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:24.011+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:24.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:24.012+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:24.868 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:24.533+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:24.868 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:24.533+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:24.869 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:24.533+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:24.869 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:24.534+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:24.869 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:24.733+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:24.869 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:24.733+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:24.869 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:24.733+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:24.869 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:24.733+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:25.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:32:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:32:24.868+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 27022 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:32:25.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:24.992+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:25.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:24.992+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:25.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:24.992+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:25.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:24.992+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:26.022 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:25.565+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:26.022 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:25.565+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:26.022 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:25.565+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:26.022 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:25.566+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:26.023 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:25.751+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:26.023 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:25.751+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:26.023 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:25.751+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:26.023 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:25.752+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:26.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:26.022+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:26.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:26.022+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:26.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:26.022+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:26.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:26.023+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:26.996 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:26.725+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:26.996 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:26.725+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:26.996 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:26.725+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:26.996 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:26.726+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:26.997 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:26.595+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:26.997 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:26.595+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:26.997 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:26.595+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:26.997 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:26.596+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:27.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:26.996+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:27.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:26.996+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:27.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:26.996+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:27.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:26.998+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:28.019 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:27.687+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:28.019 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:27.687+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:28.019 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:27.687+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:28.020 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:27.688+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:28.020 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:27.605+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:28.020 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:27.605+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:28.020 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:27.605+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:28.020 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:27.606+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:28.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:28.019+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:28.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:28.019+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:28.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:28.019+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:28.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:28.020+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:29.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:28.591+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:29.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:28.591+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:29.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:28.591+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:29.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:28.592+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:29.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:28.702+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:29.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:28.702+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:29.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:28.702+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:29.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:28.703+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:29.383 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:29.055+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:29.384 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:29.055+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:29.384 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:29.055+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:29.384 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:29.056+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:29.732 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:29.598+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:29.732 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:29.598+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:29.732 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:29.598+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:29.732 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:29.598+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:29.993 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:32:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:32:29.871+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 27103 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:32:29.993 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:29.732+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:29.993 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:29.732+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:29.994 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:29.732+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:29.994 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:29.733+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:30.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:30.045+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:30.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:30.045+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:30.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:30.045+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:30.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:30.046+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:31.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:30.637+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:31.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:30.637+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:31.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:30.637+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:31.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:30.638+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:31.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:30.751+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:31.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:30.751+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:31.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:30.751+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:31.028 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:30.752+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:31.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:31.080+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:31.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:31.080+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:31.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:31.080+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:31.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:31.081+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:32.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:31.759+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:32.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:31.759+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:32.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:31.759+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:32.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:31.760+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:32.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:31.645+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:32.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:31.645+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:32.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:31.645+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:32.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:31.646+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:32.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:32.108+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:32.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:32.108+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:32.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:32.108+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:32.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:32.109+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:33.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:32.778+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:33.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:32.778+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:33.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:32.778+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:33.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:32.779+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:33.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:32.661+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:33.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:32.661+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:33.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:32.661+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:33.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:32.662+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:33.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:33.069+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:33.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:33.069+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:33.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:33.069+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:33.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:33.070+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:34.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:33.815+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:34.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:33.815+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:34.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:33.815+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:34.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:33.816+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:34.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:33.614+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:34.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:33.614+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:34.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:33.614+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:34.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:33.614+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:34.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:34.047+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:34.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:34.047+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:34.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:34.047+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:34.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:34.048+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:34.875 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:34.816+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:34.875 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:34.816+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:34.875 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:34.816+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:34.875 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:34.817+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:34.876 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:34.612+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:34.876 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:34.612+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:34.876 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:34.612+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:34.876 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:34.613+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:35.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:32:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:32:34.875+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 27187 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:32:35.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:35.049+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:35.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:35.049+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:35.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:35.049+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:35.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:35.050+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:35.834 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:35.568+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:35.835 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:35.568+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:35.835 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:35.568+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:35.835 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:35.569+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:36.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:36.004+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:36.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:36.004+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:36.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:36.004+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:36.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:36.005+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:36.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:35.834+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:36.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:35.834+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:36.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:35.834+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:36.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:35.835+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:36.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:36.592+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:36.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:36.592+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:36.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:36.592+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:36.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:36.593+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:37.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:36.837+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:37.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:36.837+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:37.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:36.837+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:37.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:36.838+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:37.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:37.014+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:37.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:37.014+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:37.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:37.015+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:37.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:37.016+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:37.970 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:37.791+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:37.970 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:37.791+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:37.970 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:37.791+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:37.970 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:37.792+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:37.971 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:37.596+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:37.971 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:37.596+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:37.971 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:37.596+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:37.971 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:37.597+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:38.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:37.970+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:38.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:37.970+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:38.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:37.970+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:38.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:37.971+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:38.955 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:38.771+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:38.955 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:38.771+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:38.955 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:38.771+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:38.955 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:38.772+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:38.956 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:38.612+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:38.956 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:38.612+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:38.956 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:38.612+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:38.956 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:38.613+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:39.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:38.955+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:39.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:38.955+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:39.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:38.955+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:39.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:38.956+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:39.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:39.630+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:39.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:39.630+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:39.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:39.630+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:39.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:39.631+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:40.174 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:32:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:32:39.878+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 27269 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:32:40.175 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:39.776+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:40.175 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:39.776+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:40.175 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:39.776+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:40.175 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:39.777+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:40.176 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:39.986+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:40.176 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:39.986+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:40.176 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:39.986+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:40.176 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:39.987+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:40.997 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:40.644+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:40.997 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:40.644+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:40.997 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:40.644+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:40.997 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:40.645+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:40.997 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:40.737+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:40.998 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:40.737+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:40.998 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:40.737+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:40.998 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:40.738+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:41.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:40.997+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:41.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:40.997+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:41.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:40.997+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:41.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:40.998+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:41.958 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:41.625+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:41.959 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:41.625+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:41.959 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:41.625+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:41.959 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:41.625+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:41.959 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:41.693+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:41.959 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:41.693+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:41.960 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:41.693+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:41.960 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:41.694+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:42.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:41.959+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:42.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:41.959+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:42.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:41.959+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:42.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:41.959+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:42.922 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:42.668+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:42.923 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:42.668+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:42.923 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:42.668+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:42.923 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:42.669+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:42.923 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:42.611+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:42.923 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:42.611+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:42.923 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:42.611+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:42.924 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:42.612+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:43.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:42.923+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:43.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:42.923+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:43.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:42.923+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:43.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:42.923+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:43.934 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:43.685+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:43.934 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:43.685+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:43.935 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:43.685+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:43.935 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:43.686+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:43.935 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:43.655+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:43.935 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:43.655+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:43.935 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:43.655+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:43.935 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:43.656+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:44.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:43.934+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:44.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:43.934+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:44.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:43.934+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:44.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:43.935+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:44.822 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:44.681+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:44.823 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:44.681+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:44.823 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:44.681+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:44.823 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:44.681+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:44.823 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:44.655+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:44.823 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:44.655+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:44.823 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:44.655+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:44.824 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:44.656+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:45.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:32:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:32:44.881+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 27352 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:32:45.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:44.975+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:45.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:44.975+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:45.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:44.975+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:45.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:44.976+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:45.958 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:45.703+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:45.959 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:45.703+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:45.959 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:45.703+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:45.959 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:45.704+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:45.959 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:45.648+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:45.959 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:45.648+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:45.960 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:45.648+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:45.960 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:45.648+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:46.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:45.959+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:46.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:45.959+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:46.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:45.959+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:46.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:45.959+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:46.921 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:46.684+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:46.922 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:46.684+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:46.922 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:46.684+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:46.922 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:46.684+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:46.922 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:46.696+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:46.922 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:46.696+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:46.922 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:46.696+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:46.923 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:46.697+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:47.247 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:46.959+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:47.247 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:46.959+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:47.247 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:46.959+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:47.247 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:46.960+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:47.975 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:47.709+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:47.975 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:47.709+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:47.975 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:47.709+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:47.975 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:47.710+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:47.975 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:47.710+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:47.976 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:47.710+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:47.976 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:47.710+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:47.976 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:47.711+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:48.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:47.975+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:48.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:47.975+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:48.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:47.975+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:48.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:47.976+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:48.993 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:48.680+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:48.993 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:48.680+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:48.993 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:48.680+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:48.993 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:48.681+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:48.993 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:48.733+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:48.994 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:48.733+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:48.994 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:48.733+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:48.994 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:48.734+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:49.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:48.993+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:49.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:48.993+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:49.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:48.993+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:49.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:48.994+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:50.000 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:32:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:32:49.885+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 27434 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:32:50.001 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:49.706+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:50.001 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:49.706+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:50.001 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:49.706+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:50.001 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:49.707+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:50.002 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:49.722+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:50.002 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:49.722+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:50.002 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:49.722+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:50.002 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:49.723+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:50.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:50.001+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:50.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:50.001+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:50.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:50.001+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:50.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:50.002+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:51.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:50.771+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:51.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:50.771+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:51.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:50.771+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:51.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:50.772+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:51.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:50.689+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:51.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:50.689+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:51.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:50.689+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:51.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:50.689+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:51.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:51.039+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:51.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:51.039+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:51.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:51.039+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:51.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:51.040+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:52.009 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:51.803+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:52.009 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:51.803+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:52.009 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:51.803+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:52.009 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:51.804+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:52.009 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:51.657+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:52.009 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:51.657+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:52.010 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:51.657+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:52.010 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:51.658+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:52.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:52.009+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:52.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:52.009+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:52.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:52.009+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:52.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:52.010+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:53.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:52.804+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:53.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:52.804+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:53.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:52.804+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:53.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:52.805+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:53.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:52.679+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:53.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:52.679+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:53.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:52.679+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:53.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:52.680+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:53.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:53.032+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:53.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:53.032+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:53.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:53.032+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:53.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:53.032+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:54.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:53.850+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:54.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:53.850+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:54.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:53.850+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:54.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:53.851+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:54.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:53.692+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:54.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:53.692+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:54.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:53.692+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:54.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:53.693+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:54.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:54.036+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:54.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:54.036+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:54.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:54.036+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:54.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:54.037+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:55.011 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:32:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:32:54.888+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 27517 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:32:55.011 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:54.807+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:55.012 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:54.807+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:55.012 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:54.807+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:55.012 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:54.808+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:55.012 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:54.695+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:55.012 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:54.695+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:55.012 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:54.695+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:55.012 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:54.696+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:55.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:55.063+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:55.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:55.063+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:55.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:55.063+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:55.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:55.064+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:55.966 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:55.699+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:55.966 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:55.699+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:55.966 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:55.699+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:55.966 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:55.700+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:55.967 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:55.765+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:55.967 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:55.765+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:55.967 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:55.765+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:55.967 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:55.766+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:56.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:56.015+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:56.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:56.015+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:56.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:56.015+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:56.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:56.016+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:56.991 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:56.801+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:56.991 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:56.801+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:56.991 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:56.801+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:56.991 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:56.801+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:56.992 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:56.707+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:56.992 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:56.707+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:56.992 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:56.707+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:56.992 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:56.708+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:57.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:56.992+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:57.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:56.992+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:57.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:56.992+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:57.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:56.992+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:57.794 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:57.756+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:57.794 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:57.756+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:57.794 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:57.756+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:57.794 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:57.757+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:58.055 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:57.837+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:58.055 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:57.837+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:58.055 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:57.837+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:58.055 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:57.837+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:58.056 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:57.983+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:58.056 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:57.983+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:58.056 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:57.983+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:58.056 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:57.984+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:59.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:58.800+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:59.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:58.800+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:59.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:58.800+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:32:59.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:58.800+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:59.026 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:58.989+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:59.026 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:58.989+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:59.026 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:58.989+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:32:59.026 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:32:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:32:58.990+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:32:59.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:58.747+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:59.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:58.747+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:59.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:58.747+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:32:59.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:58.748+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:00.014 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:32:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:32:59.891+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 27599 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:33:00.015 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:59.771+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:00.015 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:59.771+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:00.015 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:59.771+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:00.015 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:32:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:32:59.772+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:00.016 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:59.714+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:00.016 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:59.714+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:00.016 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:59.714+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:00.016 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:32:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:32:59.715+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:00.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:00.035+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:00.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:00.035+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:00.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:00.035+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:00.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:00.035+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:00.961 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:00.692+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:00.962 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:00.692+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:00.962 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:00.692+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:00.962 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:00.692+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:00.962 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:00.743+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:00.962 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:00.743+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:00.963 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:00.743+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:00.963 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:00.744+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:01.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:01.043+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:01.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:01.043+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:01.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:01.043+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:01.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:01.044+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:02.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:01.667+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:02.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:01.667+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:02.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:01.667+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:02.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:01.668+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:02.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:01.780+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:02.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:01.780+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:02.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:01.780+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:02.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:01.781+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:02.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:02.074+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:02.540 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:02.074+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:02.540 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:02.074+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:02.540 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:02.075+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:03.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:02.709+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:03.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:02.709+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:03.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:02.709+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:03.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:02.710+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:03.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:02.757+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:03.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:02.757+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:03.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:02.757+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:03.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:02.758+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:03.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:03.035+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:03.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:03.035+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:03.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:03.035+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:03.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:03.036+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:04.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:03.731+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:04.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:03.731+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:04.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:03.731+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:04.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:03.731+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:04.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:03.747+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:04.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:03.747+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:04.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:03.747+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:04.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:03.747+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:04.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:04.052+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:04.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:04.052+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:04.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:04.052+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:04.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:04.052+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:05.018 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:33:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:33:04.894+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 27682 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:33:05.019 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:04.718+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:05.019 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:04.718+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:05.019 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:04.718+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:05.019 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:04.719+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:05.019 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:04.743+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:05.019 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:04.743+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:05.019 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:04.743+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:05.020 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:04.744+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:05.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:05.091+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:05.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:05.091+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:05.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:05.091+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:05.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:05.092+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:06.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:05.754+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:06.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:05.754+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:06.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:05.754+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:06.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:05.754+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:06.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:05.723+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:06.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:05.723+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:06.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:05.723+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:06.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:05.724+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:06.436 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:06.122+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:06.437 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:06.122+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:06.437 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:06.122+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:06.437 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:06.123+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:07.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:06.763+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:07.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:06.763+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:07.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:06.763+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:07.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:06.764+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:07.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:06.744+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:07.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:06.744+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:07.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:06.744+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:07.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:06.745+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:07.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:07.153+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:07.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:07.153+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:07.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:07.153+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:07.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:07.154+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:08.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:07.758+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:08.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:07.758+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:08.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:07.758+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:08.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:07.759+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:08.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:07.697+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:08.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:07.697+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:08.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:07.697+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:08.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:07.698+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:08.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:08.176+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:08.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:08.176+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:08.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:08.176+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:08.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:08.176+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:09.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:08.698+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:09.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:08.698+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:09.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:08.698+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:09.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:08.699+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:09.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:08.742+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:09.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:08.742+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:09.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:08.742+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:09.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:08.743+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:09.383 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:09.187+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:09.384 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:09.187+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:09.384 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:09.187+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:09.384 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:09.188+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:10.019 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:33:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:33:09.898+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 27764 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:33:10.019 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:09.707+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:10.020 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:09.707+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:10.020 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:09.707+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:10.020 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:09.708+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:10.020 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:09.733+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:10.020 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:09.733+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:10.020 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:09.733+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:10.021 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:09.734+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:10.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:10.225+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:10.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:10.225+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:10.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:10.225+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:10.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:10.226+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:11.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:10.715+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:11.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:10.715+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:11.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:10.715+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:11.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:10.715+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:11.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:10.728+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:11.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:10.728+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:11.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:10.728+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:11.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:10.729+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:11.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:11.186+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:11.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:11.186+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:11.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:11.186+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:11.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:11.187+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:12.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:11.718+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:12.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:11.718+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:12.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:11.718+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:12.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:11.719+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:12.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:11.714+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:12.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:11.714+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:12.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:11.714+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:12.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:11.715+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:12.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:12.152+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:12.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:12.152+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:12.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:12.152+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:12.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:12.153+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:13.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:12.742+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:13.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:12.742+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:13.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:12.742+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:13.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:12.743+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:13.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:12.736+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:13.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:12.736+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:13.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:12.736+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:13.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:12.737+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:13.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:13.169+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:13.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:13.169+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:13.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:13.169+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:13.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:13.170+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:14.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:13.778+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:14.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:13.778+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:14.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:13.778+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:14.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:13.779+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:14.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:13.755+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:14.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:13.755+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:14.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:13.755+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:14.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:13.755+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:14.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:14.216+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:14.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:14.216+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:14.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:14.216+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:14.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:14.217+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:15.207 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:33:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:33:14.901+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 27847 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:33:15.208 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:14.802+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:15.208 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:14.802+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:15.208 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:14.802+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:15.208 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:14.803+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:15.208 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:14.792+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:15.209 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:14.792+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:15.209 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:14.792+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:15.209 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:14.793+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:15.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:15.208+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:15.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:15.208+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:15.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:15.208+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:15.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:15.209+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:16.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:15.761+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:16.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:15.761+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:16.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:15.761+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:16.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:15.762+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:16.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:15.817+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:16.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:15.817+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:16.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:15.817+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:16.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:15.818+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:16.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:16.251+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:16.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:16.251+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:16.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:16.251+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:16.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:16.251+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:17.274 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:16.779+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:17.274 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:16.779+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:17.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:16.779+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:17.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:16.780+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:17.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:16.852+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:17.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:16.852+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:17.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:16.852+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:17.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:16.853+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:17.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:17.274+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:17.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:17.274+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:17.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:17.274+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:17.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:17.276+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:18.257 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:17.805+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:18.257 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:17.805+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:18.257 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:17.805+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:18.257 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:17.806+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:18.257 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:17.878+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:18.258 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:17.878+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:18.258 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:17.878+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:18.258 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:17.879+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:18.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:18.257+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:18.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:18.257+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:18.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:18.257+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:18.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:18.258+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:19.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:18.850+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:19.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:18.850+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:19.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:18.850+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:19.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:18.851+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:19.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:18.843+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:19.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:18.843+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:19.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:18.843+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:19.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:18.844+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:19.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:19.287+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:19.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:19.287+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:19.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:19.287+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:19.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:19.288+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:20.176 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:33:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:33:19.904+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 27928 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:33:20.177 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:19.813+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:20.177 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:19.813+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:20.177 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:19.813+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:20.177 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:19.813+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:20.177 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:19.859+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:20.177 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:19.859+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:20.178 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:19.859+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:20.178 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:19.859+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:20.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:20.327+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:20.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:20.327+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:20.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:20.327+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:20.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:20.327+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:21.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:20.799+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:21.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:20.799+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:21.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:20.799+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:21.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:20.799+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:21.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:20.866+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:21.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:20.866+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:21.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:20.866+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:21.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:20.867+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:21.744 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:21.308+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:21.745 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:21.308+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:21.745 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:21.308+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:21.745 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:21.309+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:22.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:21.835+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:22.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:21.835+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:22.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:21.835+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:22.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:21.836+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:22.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:21.896+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:22.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:21.897+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:22.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:21.897+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:22.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:21.897+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:22.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:22.293+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:22.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:22.293+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:22.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:22.293+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:22.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:22.294+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:23.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:22.859+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:23.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:22.859+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:23.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:22.859+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:23.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:22.860+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:23.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:22.926+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:23.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:22.926+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:23.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:22.926+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:23.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:22.927+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:23.627 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:23.316+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:23.628 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:23.316+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:23.628 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:23.316+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:23.628 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:23.317+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:24.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:23.906+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:24.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:23.906+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:24.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:23.906+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:24.277 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:23.906+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:24.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:23.900+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:24.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:23.900+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:24.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:23.900+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:24.278 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:23.900+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:24.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:24.311+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:24.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:24.311+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:24.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:24.311+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:24.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:24.312+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:25.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:33:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:33:24.907+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 28011 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:33:25.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:24.869+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:25.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:24.869+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:25.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:24.869+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:25.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:24.870+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:25.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:24.883+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:25.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:24.883+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:25.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:24.883+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:25.277 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:24.884+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:25.537 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:25.287+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:25.538 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:25.287+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:25.538 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:25.287+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:25.538 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:25.288+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:26.247 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:25.895+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:26.247 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:25.895+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:26.247 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:25.895+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:26.247 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:25.895+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:26.248 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:25.826+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:26.248 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:25.826+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:26.248 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:25.826+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:26.248 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:25.827+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:26.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:26.247+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:26.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:26.247+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:26.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:26.247+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:26.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:26.248+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:27.267 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:26.868+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:27.267 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:26.868+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:27.267 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:26.868+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:27.267 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:26.869+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:27.267 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:26.836+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:27.267 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:26.836+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:27.268 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:26.836+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:27.268 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:26.836+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:27.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:27.267+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:27.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:27.267+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:27.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:27.267+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:27.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:27.268+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:28.265 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:27.861+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:28.265 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:27.861+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:28.265 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:27.861+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:28.265 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:27.862+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:28.265 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:27.909+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:28.266 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:27.909+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:28.266 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:27.909+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:28.266 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:27.910+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:28.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:28.265+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:28.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:28.265+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:28.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:28.265+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:28.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:28.266+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:29.269 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:28.846+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:29.270 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:28.846+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:29.270 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:28.846+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:29.270 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:28.847+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:29.270 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:28.934+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:29.270 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:28.934+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:29.270 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:28.934+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:29.271 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:28.935+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:29.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:29.270+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:29.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:29.270+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:29.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:29.270+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:29.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:29.270+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:30.177 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:33:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:33:29.911+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 28093 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:33:30.178 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:29.920+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:30.178 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:29.920+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:30.178 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:29.920+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:30.178 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:29.921+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:30.178 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:29.879+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:30.178 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:29.879+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:30.179 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:29.879+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:30.179 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:29.880+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:30.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:30.243+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:30.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:30.243+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:30.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:30.243+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:30.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:30.243+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:31.210 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:30.921+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:31.210 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:30.921+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:31.210 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:30.921+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:31.210 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:30.922+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:31.211 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:30.902+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:31.211 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:30.902+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:31.211 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:30.902+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:31.211 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:30.903+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:31.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:31.210+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:31.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:31.210+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:31.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:31.210+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:31.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:31.211+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:31.964 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:31.857+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:31.964 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:31.857+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:31.964 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:31.857+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:31.965 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:31.858+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:32.239 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:31.965+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:32.240 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:31.965+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:32.240 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:31.965+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:32.240 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:31.966+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:32.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:32.240+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:32.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:32.240+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:32.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:32.240+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:32.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:32.241+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:33.264 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:32.979+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:33.264 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:32.979+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:33.264 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:32.979+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:33.264 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:32.981+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:33.264 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:32.832+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:33.265 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:32.832+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:33.265 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:32.832+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:33.265 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:32.833+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:33.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:33.264+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:33.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:33.264+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:33.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:33.264+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:33.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:33.265+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:33.936 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:33.849+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:33.936 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:33.849+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:33.936 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:33.849+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:33.936 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:33.849+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:34.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:33.936+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:34.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:33.936+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:34.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:33.936+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:34.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:33.937+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:34.753 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:34.292+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:34.754 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:34.292+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:34.754 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:34.292+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:34.754 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:34.293+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:35.025 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:33:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:33:34.914+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 28177 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:33:35.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:34.896+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:35.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:34.896+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:35.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:34.896+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:35.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:34.896+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:35.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:34.813+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:35.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:34.813+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:35.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:34.813+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:35.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:34.814+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:35.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:35.269+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:35.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:35.269+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:35.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:35.269+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:35.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:35.270+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:36.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:35.942+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:36.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:35.942+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:36.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:35.942+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:36.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:35.943+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:36.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:35.785+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:36.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:35.785+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:36.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:35.785+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:36.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:35.786+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:36.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:36.281+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:36.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:36.281+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:36.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:36.281+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:36.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:36.281+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:37.246 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:36.972+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:37.246 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:36.972+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:37.246 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:36.972+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:37.246 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:36.973+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:37.247 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:36.789+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:37.247 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:36.789+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:37.247 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:36.789+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:37.247 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:36.790+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:37.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:37.246+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:37.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:37.246+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:37.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:37.246+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:37.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:37.247+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:38.269 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:37.941+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:38.270 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:37.941+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:38.270 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:37.941+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:38.271 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:37.942+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:38.271 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:37.839+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:38.271 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:37.839+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:38.271 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:37.839+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:38.272 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:37.840+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:38.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:38.267+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:38.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:38.267+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:38.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:38.267+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:38.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:38.268+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:39.262 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:38.939+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:39.262 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:38.939+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:39.262 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:38.939+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:39.262 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:38.940+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:39.263 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:38.878+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:39.263 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:38.878+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:39.263 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:38.878+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:39.263 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:38.879+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:39.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:39.262+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:39.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:39.262+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:39.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:39.262+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:39.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:39.263+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:40.178 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:33:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:33:39.917+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 28259 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:33:40.179 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:39.898+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:40.179 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:39.898+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:40.179 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:39.898+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:40.179 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:39.898+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:40.179 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:39.878+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:40.180 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:39.878+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:40.180 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:39.878+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:40.180 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:39.879+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:40.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:40.228+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:40.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:40.228+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:40.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:40.228+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:40.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:40.229+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:41.198 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:40.865+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:41.199 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:40.865+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:41.199 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:40.865+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:41.199 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:40.866+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:41.199 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:40.890+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:41.199 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:40.890+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:41.200 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:40.890+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:41.200 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:40.891+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:41.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:41.198+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:41.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:41.198+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:41.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:41.198+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:41.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:41.199+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:42.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:41.857+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:42.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:41.857+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:42.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:41.857+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:42.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:41.858+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:42.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:41.888+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:42.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:41.888+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:42.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:41.888+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:42.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:41.889+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:42.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:42.162+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:42.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:42.162+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:42.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:42.162+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:42.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:42.162+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:43.167 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:42.865+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:43.167 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:42.865+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:43.167 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:42.865+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:43.168 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:42.866+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:43.168 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:42.860+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:43.168 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:42.860+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:43.168 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:42.860+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:43.168 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:42.861+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:43.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:43.168+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:43.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:43.168+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:43.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:43.168+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:43.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:43.168+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:43.969 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:43.825+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:43.969 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:43.825+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:43.969 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:43.825+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:43.970 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:43.826+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:43.970 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:43.851+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:43.970 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:43.851+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:43.970 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:43.851+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:43.970 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:43.852+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:44.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:44.143+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:44.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:44.143+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:44.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:44.143+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:44.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:44.144+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:45.094 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:44.836+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:45.095 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:44.836+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:45.095 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:44.836+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:45.095 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:44.836+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:45.095 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:44.829+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:45.095 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:44.829+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:45.095 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:44.829+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:45.096 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:44.830+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:45.096 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:33:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:33:44.920+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 28342 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:33:45.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:45.095+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:45.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:45.095+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:45.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:45.095+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:45.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:45.096+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:46.066 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:45.803+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:46.066 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:45.803+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:46.066 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:45.804+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:46.067 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:45.804+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:46.067 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:45.795+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:46.067 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:45.795+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:46.067 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:45.795+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:46.067 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:45.796+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:46.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:46.066+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:46.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:46.066+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:46.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:46.066+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:46.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:46.067+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:47.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:46.771+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:47.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:46.771+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:47.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:46.771+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:47.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:46.772+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:47.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:46.828+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:47.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:46.828+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:47.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:46.828+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:47.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:46.828+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:47.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:47.115+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:47.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:47.115+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:47.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:47.115+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:47.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:47.116+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:48.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:47.774+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:48.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:47.774+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:48.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:47.774+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:48.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:47.775+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:48.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:47.843+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:48.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:47.843+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:48.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:47.843+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:48.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:47.844+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:48.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:48.103+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:48.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:48.103+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:48.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:48.103+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:48.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:48.104+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:49.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:48.740+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:49.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:48.740+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:49.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:48.740+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:49.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:48.741+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:49.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:48.848+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:49.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:48.848+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:49.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:48.848+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:49.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:48.848+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:49.383 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:49.098+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:49.383 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:49.098+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:49.383 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:49.098+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:49.383 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:49.099+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:50.025 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:33:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:33:49.924+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 28426 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:33:50.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:49.821+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:50.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:49.821+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:50.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:49.821+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:50.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:49.822+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:50.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:49.734+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:50.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:49.734+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:50.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:49.734+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:50.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:49.735+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:50.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:50.066+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:50.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:50.066+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:50.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:50.066+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:50.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:50.066+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:51.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:50.754+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:51.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:50.754+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:51.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:50.754+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:51.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:50.755+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:51.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:50.799+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:51.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:50.799+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:51.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:50.799+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:51.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:50.800+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:51.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:51.052+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:51.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:51.052+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:51.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:51.052+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:51.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:51.052+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:52.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:51.761+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:52.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:51.761+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:52.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:51.761+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:52.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:51.762+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:52.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:51.795+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:52.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:51.795+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:52.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:51.795+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:52.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:51.796+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:52.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:52.100+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:52.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:52.100+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:52.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:52.100+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:52.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:52.101+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:53.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:52.774+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:53.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:52.774+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:53.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:52.774+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:53.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:52.775+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:53.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:52.786+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:53.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:52.786+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:53.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:52.786+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:53.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:52.787+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:53.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:53.142+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:53.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:53.142+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:53.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:53.142+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:53.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:53.143+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:54.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:53.806+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:54.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:53.806+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:54.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:53.806+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:54.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:53.807+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:54.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:53.746+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:54.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:53.746+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:54.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:53.746+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:54.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:53.747+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:54.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:54.143+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:54.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:54.143+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:54.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:54.143+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:54.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:54.144+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:55.025 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:33:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:33:54.927+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 28509 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:33:55.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:54.714+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:55.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:54.714+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:55.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:54.714+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:55.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:54.715+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:55.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:54.797+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:55.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:54.797+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:55.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:54.797+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:55.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:54.797+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:55.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:55.161+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:55.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:55.161+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:55.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:55.161+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:55.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:55.162+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:56.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:55.797+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:56.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:55.797+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:56.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:55.797+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:56.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:55.798+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:56.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:55.665+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:56.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:55.665+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:56.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:55.665+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:22.315455+0000 (oldest deadline 2024-03-11T23:23:42.315455+0000) 2024-03-11T23:33:56.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:55.666+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:56.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:56.203+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:56.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:56.203+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:56.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:56.203+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:56.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:56.204+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:57.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:56.696+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:57.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:56.823+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:57.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:56.823+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:57.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:56.823+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:57.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:56.824+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:57.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:57.238+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:57.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:57.238+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:57.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:57.238+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:57.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:57.240+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:58.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:57.715+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:58.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:57.850+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:58.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:57.850+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:58.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:57.850+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:58.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:57.850+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:58.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:58.256+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:58.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:58.256+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:58.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:58.256+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:58.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:58.257+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:59.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:58.684+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:59.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:58.899+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:59.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:58.899+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:59.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:58.899+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:59.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:58.900+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:59.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:59.250+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:59.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:59.251+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:59.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:59.251+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:33:59.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:33:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:33:59.251+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:59.931 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:33:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:33:59.649+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:33:59.931 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:59.869+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:59.931 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:59.869+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:59.931 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:59.869+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:33:59.932 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:33:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:33:59.870+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:00.255 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:33:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:33:59.931+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 28591 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:34:00.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:00.256+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:00.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:00.256+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:00.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:00.256+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:00.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:00.257+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:00.911 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:00.656+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:01.237 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:00.912+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:34:01.238 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:00.912+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:34:01.238 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:00.912+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:34:01.238 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:00.913+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:01.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:01.238+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:01.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:01.238+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:01.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:01.238+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:01.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:01.239+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:01.885 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:01.620+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:02.244 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:01.886+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:34:02.245 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:01.886+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:34:02.245 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:01.886+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:34:02.245 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:01.887+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:02.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:02.245+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:02.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:02.245+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:02.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:02.245+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:02.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:02.246+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:02.933 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:02.656+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:03.208 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:02.934+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:34:03.208 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:02.934+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:34:03.208 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:02.934+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:34:03.209 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:02.935+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:03.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:03.209+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:03.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:03.209+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:03.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:03.209+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:03.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:03.210+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:03.932 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:03.622+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:04.206 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:03.932+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:34:04.206 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:03.932+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:34:04.206 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:03.932+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:34:04.206 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:03.933+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:04.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:04.206+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:04.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:04.206+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:04.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:04.206+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:04.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:04.207+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:04.915 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:04.594+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:05.237 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:34:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:34:04.935+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 28674 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:34:05.237 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:04.916+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:34:05.237 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:04.916+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:34:05.238 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:04.916+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:35.829192+0000 (oldest deadline 2024-03-11T23:23:55.829192+0000) 2024-03-11T23:34:05.238 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:04.917+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:05.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:05.237+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:05.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:05.237+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:05.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:05.237+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:05.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:05.238+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:05.892 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:05.574+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:06.202 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:05.893+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:06.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:06.203+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:06.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:06.203+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:06.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:06.203+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:06.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:06.203+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:06.886 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:06.583+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:07.238 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:06.886+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:07.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:07.238+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:07.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:07.238+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:07.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:07.238+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:07.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:07.239+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:07.881 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:07.623+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:08.207 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:07.882+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:08.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:08.208+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:08.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:08.208+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:08.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:08.208+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:08.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:08.209+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:08.924 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:08.603+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:09.205 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:08.925+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:09.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:09.205+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:09.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:09.205+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:09.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:09.205+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:09.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:09.206+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:09.937 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:09.565+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:10.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:34:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:34:09.938+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 28757 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:34:10.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:09.936+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:10.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:10.174+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:10.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:10.174+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:10.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:10.174+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:10.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:10.175+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:10.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:10.525+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:11.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:10.934+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:11.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:11.161+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:11.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:11.161+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:11.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:11.161+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:11.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:11.162+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:11.951 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:11.551+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:12.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:11.951+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:12.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:12.127+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:12.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:12.127+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:12.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:12.127+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:12.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:12.127+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:12.970 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:12.527+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:13.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:12.971+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:13.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:13.166+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:13.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:13.166+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:13.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:13.166+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:23:38.427537+0000 (oldest deadline 2024-03-11T23:23:58.427537+0000) 2024-03-11T23:34:13.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:13.166+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:13.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:13.479+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:14.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:14.003+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:14.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:14.127+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:14.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:14.475+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:15.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:34:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:34:14.942+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 28840 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:34:15.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:14.993+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:15.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:15.149+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:15.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:15.471+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:16.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:16.015+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:16.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:16.143+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:16.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:16.479+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:17.504 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:17.041+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:17.504 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:17.104+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:17.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:17.504+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:18.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:18.032+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:18.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:18.064+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:19.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:18.533+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:19.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:18.533+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:19.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:18.534+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:19.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:18.534+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:19.383 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:19.038+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:19.384 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:19.088+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:19.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:19.497+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:19.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:19.497+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:19.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:19.497+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:19.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:19.498+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:20.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:34:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:34:19.945+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 28922 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:34:20.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:20.031+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:20.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:20.083+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:20.833 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:20.532+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:20.834 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:20.532+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:20.834 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:20.532+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:20.834 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:20.533+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:21.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:21.004+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:21.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:21.063+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:22.013 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:21.550+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:22.014 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:21.550+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:22.014 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:21.550+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:22.014 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:21.551+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:22.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:22.099+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:22.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:22.014+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:22.981 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:22.534+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:22.982 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:22.534+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:22.982 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:22.534+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:22.982 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:22.535+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:23.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:22.982+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:23.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:23.053+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:23.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:23.519+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:23.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:23.519+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:23.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:23.519+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:23.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:23.519+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:24.278 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:23.986+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:24.278 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:24.030+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:24.948 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:24.545+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:24.948 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:24.545+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:24.949 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:24.545+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:24.949 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:24.545+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:25.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:34:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:34:24.949+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 29005 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:34:25.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:25.001+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:25.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:25.009+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:25.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:25.522+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:25.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:25.522+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:25.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:25.522+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:25.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:25.524+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:26.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:25.997+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:26.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:26.030+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:27.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:26.542+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:27.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:26.542+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:27.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:26.542+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:27.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:26.543+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:27.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:27.053+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:27.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:27.034+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:27.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:27.034+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:27.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:27.034+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:27.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:27.035+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:27.991 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:27.574+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:27.991 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:27.574+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:27.992 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:27.574+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:27.992 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:27.575+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:28.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:28.065+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:28.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:27.992+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:28.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:27.992+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:28.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:27.992+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:28.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:27.993+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:28.963 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:28.587+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:28.963 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:28.587+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:28.963 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:28.587+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:28.964 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:28.588+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:29.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:29.069+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:29.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:28.964+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:29.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:28.964+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:29.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:28.964+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:29.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:28.965+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:29.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:29.580+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:29.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:29.580+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:29.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:29.580+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:29.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:29.581+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:30.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:34:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:34:29.952+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 29088 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:34:30.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:30.105+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:30.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:29.955+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:30.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:29.955+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:30.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:29.955+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:30.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:29.956+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:30.949 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:30.624+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:30.949 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:30.624+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:30.949 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:30.624+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:30.949 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:30.624+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:31.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:31.073+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:31.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:30.949+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:31.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:30.949+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:31.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:30.949+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:31.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:30.950+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:31.990 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:31.624+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:31.990 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:31.624+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:31.990 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:31.624+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:31.990 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:31.625+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:32.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:32.116+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:32.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:31.991+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:32.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:31.991+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:32.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:31.991+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:32.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:31.992+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:32.964 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:32.593+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:32.964 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:32.593+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:32.964 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:32.593+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:32.965 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:32.594+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:33.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:33.074+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:33.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:32.964+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:33.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:32.964+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:33.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:32.964+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:33.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:32.965+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:33.935 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:33.593+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:33.935 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:33.593+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:33.935 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:33.593+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:33.936 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:33.594+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:34.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:34.099+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:34.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:33.935+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:34.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:33.935+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:34.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:33.935+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:34.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:33.936+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:34.895 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:34.629+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:34.895 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:34.629+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:34.896 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:34.629+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:34.896 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:34.630+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:35.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:35.086+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:35.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:34:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:34:34.956+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 29171 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:34:35.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:34.896+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:35.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:34.896+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:35.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:34.896+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:35.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:34.897+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:35.882 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:35.621+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:35.882 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:35.621+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:35.882 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:35.621+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:35.882 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:35.622+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:36.193 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:36.096+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:36.194 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:35.882+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:36.194 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:35.882+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:36.194 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:35.882+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:36.194 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:35.883+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:37.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:36.621+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:37.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:36.621+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:37.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:36.621+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:37.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:36.622+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:37.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:36.844+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:37.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:36.844+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:37.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:36.844+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:37.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:36.845+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:37.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:37.099+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:37.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:37.099+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:37.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:37.099+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:37.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:37.100+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:38.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:37.609+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:38.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:37.609+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:38.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:37.609+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:38.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:37.609+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:38.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:37.815+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:38.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:37.815+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:38.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:37.815+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:38.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:37.816+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:38.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:38.095+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:38.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:38.095+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:38.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:38.095+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:38.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:38.096+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:39.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:38.612+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:39.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:38.612+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:39.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:38.612+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:39.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:38.612+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:39.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:38.811+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:39.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:38.811+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:39.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:38.811+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:39.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:38.812+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:39.383 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:39.110+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:39.383 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:39.110+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:39.383 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:39.110+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:39.383 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:39.110+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:39.767 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:39.571+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:39.767 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:39.571+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:39.767 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:39.571+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:39.768 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:39.572+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:40.025 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:34:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:34:39.959+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 29254 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:34:40.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:39.768+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:40.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:39.768+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:40.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:39.768+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:40.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:39.769+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:40.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:40.099+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:40.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:40.099+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:40.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:40.099+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:40.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:40.100+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:41.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:40.733+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:41.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:40.733+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:41.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:40.733+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:41.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:40.734+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:41.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:40.574+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:41.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:40.574+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:41.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:40.574+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:41.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:40.575+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:41.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:41.091+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:41.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:41.091+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:41.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:41.091+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:41.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:41.092+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:42.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:41.725+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:42.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:41.725+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:42.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:41.725+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:42.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:41.726+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:42.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:41.543+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:42.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:41.543+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:42.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:41.543+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:42.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:41.544+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:42.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:42.128+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:42.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:42.128+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:42.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:42.128+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:42.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:42.129+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:43.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:42.569+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:43.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:42.569+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:43.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:42.569+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:43.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:42.570+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:43.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:42.691+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:43.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:42.691+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:43.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:42.691+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:43.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:42.692+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:43.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:43.174+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:43.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:43.174+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:43.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:43.174+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:43.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:43.175+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:44.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:43.739+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:44.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:43.739+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:44.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:43.739+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:44.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:43.740+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:44.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:43.599+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:44.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:43.599+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:44.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:43.599+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:44.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:43.600+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:44.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:44.176+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:44.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:44.176+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:44.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:44.176+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:44.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:44.176+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:44.962 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:44.778+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:44.962 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:44.778+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:44.962 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:44.778+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:44.962 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:44.778+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:44.963 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:44.576+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:44.963 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:44.576+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:44.963 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:44.576+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:44.963 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:44.577+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:45.222 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:34:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:34:44.962+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 29337 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:34:45.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:45.223+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:45.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:45.223+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:45.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:45.223+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:45.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:45.224+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:46.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:45.745+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:46.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:45.745+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:46.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:45.745+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:46.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:45.746+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:46.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:45.602+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:46.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:45.602+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:46.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:45.602+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:46.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:45.603+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:46.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:46.215+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:46.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:46.215+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:46.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:46.215+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:46.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:46.217+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:47.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:46.729+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:47.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:46.729+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:47.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:46.729+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:47.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:46.730+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:47.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:46.558+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:47.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:46.558+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:47.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:46.558+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:47.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:46.559+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:47.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:47.187+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:47.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:47.187+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:47.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:47.187+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:47.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:47.188+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:48.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:47.685+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:48.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:47.685+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:48.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:47.685+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:48.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:47.686+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:48.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:47.572+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:48.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:47.572+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:48.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:47.572+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:48.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:47.572+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:48.524 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:48.187+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:48.524 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:48.187+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:48.524 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:48.187+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:48.524 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:48.188+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:48.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:48.654+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:48.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:48.655+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:48.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:48.655+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:48.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:48.656+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:48.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:48.524+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:48.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:48.524+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:48.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:48.524+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:48.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:48.525+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:49.480 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:49.172+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:49.481 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:49.172+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:49.481 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:49.172+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:49.481 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:49.173+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:49.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:49.704+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:49.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:49.704+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:49.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:49.704+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:49.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:49.705+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:49.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:49.481+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:49.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:49.481+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:49.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:49.481+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:49.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:49.482+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:50.276 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:34:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:34:49.966+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 29420 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:34:50.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:50.139+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:50.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:50.139+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:50.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:50.139+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:50.277 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:50.140+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:50.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:50.729+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:50.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:50.729+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:50.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:50.729+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:50.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:50.730+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:50.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:50.525+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:50.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:50.525+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:50.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:50.525+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:50.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:50.526+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:51.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:51.164+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:51.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:51.164+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:51.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:51.164+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:51.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:51.165+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:52.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:51.708+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:52.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:51.708+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:52.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:51.708+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:52.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:51.709+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:52.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:51.536+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:52.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:51.536+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:52.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:51.536+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:52.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:51.537+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:52.519 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:52.162+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:52.519 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:52.162+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:52.519 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:52.162+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:52.519 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:52.163+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:52.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:52.685+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:52.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:52.685+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:52.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:52.685+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:52.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:52.685+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:52.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:52.519+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:52.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:52.519+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:52.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:52.519+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:52.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:52.520+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:53.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:53.147+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:53.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:53.147+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:53.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:53.147+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:53.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:53.148+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:54.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:53.672+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:54.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:53.672+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:54.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:53.672+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:54.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:53.673+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:54.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:53.558+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:54.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:53.558+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:54.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:53.558+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:54.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:53.559+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:54.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:54.104+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:54.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:54.104+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:54.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:54.104+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:54.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:54.105+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:54.969 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:54.549+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:54.969 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:54.549+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:54.969 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:54.549+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:54.969 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:54.550+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:54.970 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:54.681+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:54.970 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:54.681+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:54.970 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:54.681+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:54.970 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:54.682+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:55.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:34:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:34:54.969+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 29503 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:34:55.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:55.144+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:55.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:55.144+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:55.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:55.144+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:55.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:55.144+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:56.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:55.658+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:56.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:55.658+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:56.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:55.658+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:56.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:55.659+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:56.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:55.573+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:56.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:55.573+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:56.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:55.573+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:56.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:55.574+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:56.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:56.099+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:56.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:56.099+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:56.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:56.099+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:56.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:56.100+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:57.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:56.649+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:57.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:56.649+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:57.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:56.649+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:57.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:56.650+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:57.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:56.604+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:57.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:56.604+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:57.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:56.604+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:57.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:56.605+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:57.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:57.120+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:57.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:57.120+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:57.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:57.120+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:57.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:57.121+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:58.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:57.689+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:58.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:57.689+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:58.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:57.689+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:58.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:57.689+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:58.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:57.604+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:58.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:57.604+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:58.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:57.604+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:58.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:57.605+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:58.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:58.151+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:58.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:58.151+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:58.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:58.151+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:58.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:58.151+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:59.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:58.578+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:59.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:58.578+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:59.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:58.578+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:59.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:58.579+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:59.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:58.668+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:59.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:58.668+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:59.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:58.668+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:59.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:58.669+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:59.383 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:59.127+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:59.383 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:59.127+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:59.383 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:59.127+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:34:59.384 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:34:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:34:59.128+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:59.636 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:59.614+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:59.636 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:59.614+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:59.636 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:59.614+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:34:59.637 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:34:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:34:59.615+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:34:59.972 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:59.637+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:59.973 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:59.637+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:59.973 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:59.637+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:34:59.973 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:34:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:34:59.637+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:00.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:34:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:34:59.973+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 29586 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:35:00.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:00.144+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:00.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:00.144+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:00.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:00.144+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:00.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:00.145+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:01.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:00.620+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:01.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:00.620+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:01.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:00.620+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:01.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:00.621+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:01.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:00.644+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:01.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:00.644+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:01.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:00.644+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:01.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:00.645+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:01.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:01.130+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:01.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:01.130+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:01.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:01.130+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:01.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:01.131+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:02.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:01.644+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:02.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:01.644+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:02.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:01.644+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:02.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:01.645+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:02.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:01.636+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:02.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:01.636+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:02.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:01.636+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:02.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:01.637+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:02.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:02.163+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:02.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:02.163+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:02.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:02.163+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:02.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:02.164+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:03.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:02.672+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:03.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:02.672+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:03.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:02.672+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:03.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:02.673+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:03.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:02.644+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:03.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:02.644+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:03.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:02.645+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:03.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:02.645+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:03.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:03.143+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:03.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:03.143+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:03.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:03.143+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:03.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:03.144+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:04.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:03.716+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:04.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:03.716+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:04.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:03.716+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:04.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:03.717+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:04.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:03.612+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:04.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:03.612+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:04.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:03.612+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:04.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:03.613+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:04.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:04.165+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:04.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:04.165+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:04.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:04.165+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:04.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:04.166+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:04.976 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:04.736+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:04.976 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:04.736+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:04.977 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:04.736+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:04.977 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:04.737+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:04.977 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:04.648+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:04.977 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:04.648+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:04.977 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:04.648+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:04.977 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:04.649+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:05.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:35:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:35:04.976+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 29669 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:35:05.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:05.149+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:05.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:05.149+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:05.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:05.149+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:05.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:05.150+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:06.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:05.679+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:06.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:05.679+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:06.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:05.679+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:06.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:05.680+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:06.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:05.734+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:06.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:05.734+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:06.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:05.734+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:06.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:05.735+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:06.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:06.118+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:06.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:06.118+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:06.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:06.118+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:06.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:06.119+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:07.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:06.741+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:07.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:06.741+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:07.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:06.741+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:07.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:06.742+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:07.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:06.689+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:07.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:06.689+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:07.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:06.689+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:07.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:06.690+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:07.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:07.169+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:07.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:07.169+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:07.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:07.169+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:07.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:07.170+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:08.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:07.680+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:08.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:07.680+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:08.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:07.680+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:08.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:07.681+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:08.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:07.755+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:08.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:07.755+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:08.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:07.755+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:08.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:07.756+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:08.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:08.191+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:08.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:08.191+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:08.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:08.191+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:08.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:08.191+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:09.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:08.700+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:09.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:08.700+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:09.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:08.700+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:09.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:08.701+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:09.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:08.748+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:09.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:08.748+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:09.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:08.748+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:09.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:08.749+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:09.383 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:09.160+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:09.383 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:09.160+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:09.383 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:09.160+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:09.383 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:09.160+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:09.980 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:09.713+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:09.980 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:09.713+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:09.980 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:09.713+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:09.981 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:09.714+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:09.981 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:09.661+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:09.981 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:09.661+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:09.981 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:09.661+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:09.981 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:09.662+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:10.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:35:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:35:09.980+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 29752 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:35:10.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:10.187+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:10.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:10.187+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:10.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:10.187+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:10.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:10.188+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:11.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:10.706+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:11.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:10.706+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:11.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:10.706+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:11.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:10.707+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:11.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:10.688+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:11.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:10.688+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:11.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:10.688+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:11.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:10.689+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:11.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:11.173+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:11.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:11.173+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:11.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:11.173+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:11.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:11.174+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:12.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:11.647+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:12.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:11.647+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:12.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:11.647+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:12.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:11.647+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:12.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:11.731+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:12.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:11.731+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:12.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:11.731+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:12.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:11.731+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:12.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:12.169+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:12.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:12.169+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:12.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:12.169+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:12.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:12.169+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:13.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:12.777+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:13.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:12.777+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:13.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:12.777+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:13.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:12.777+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:13.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:12.608+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:13.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:12.608+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:13.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:12.608+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:13.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:12.608+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:13.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:13.122+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:13.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:13.122+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:13.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:13.122+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:13.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:13.123+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:14.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:13.797+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:14.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:13.797+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:14.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:13.797+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:14.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:13.797+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:14.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:13.584+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:14.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:13.584+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:14.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:13.584+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:14.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:13.585+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:14.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:14.078+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:14.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:14.078+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:14.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:14.078+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:14.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:14.078+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:14.983 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:14.557+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:14.984 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:14.557+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:14.984 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:14.557+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:14.984 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:14.557+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:14.984 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:14.756+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:14.984 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:14.756+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:14.984 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:14.756+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:14.984 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:14.757+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:15.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:35:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:35:14.983+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 29835 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:35:15.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:15.117+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:15.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:15.117+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:15.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:15.117+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:15.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:15.118+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:15.801 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:15.537+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:15.802 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:15.537+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:15.802 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:15.537+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:15.802 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:15.538+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:16.101 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:15.802+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:16.101 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:15.802+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:16.101 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:15.802+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:16.102 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:15.803+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:16.518 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:16.102+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:16.519 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:16.102+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:16.519 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:16.102+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:16.519 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:16.103+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:16.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:16.519+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:16.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:16.519+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:16.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:16.519+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:16.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:16.520+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:17.052 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:16.798+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:17.052 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:16.798+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:17.052 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:16.798+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:17.052 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:16.798+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:17.470 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:17.054+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:17.471 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:17.054+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:17.471 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:17.054+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:17.471 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:17.054+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:17.764 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:17.472+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:17.765 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:17.472+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:17.765 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:17.472+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:17.765 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:17.473+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:18.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:17.765+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:18.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:17.765+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:18.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:17.765+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:18.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:17.766+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:18.470 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:18.073+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:18.471 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:18.073+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:18.471 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:18.073+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:18.471 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:18.073+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:18.752 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:18.471+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:18.752 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:18.471+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:18.752 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:18.471+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:18.752 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:18.472+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:19.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:18.752+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:19.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:18.752+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:19.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:18.752+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:19.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:18.753+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:19.382 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:19.058+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:19.383 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:19.058+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:19.383 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:19.058+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:19.383 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:19.058+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:19.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:19.507+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:19.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:19.507+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:19.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:19.507+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:19.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:19.508+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:20.038 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:35:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:35:19.987+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 29912 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:35:20.038 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:19.778+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:20.038 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:19.778+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:20.038 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:19.778+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:20.039 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:19.779+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:20.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:20.038+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:20.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:20.038+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:20.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:20.038+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:20.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:20.039+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:20.826 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:20.530+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:20.826 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:20.530+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:20.826 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:20.530+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:20.826 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:20.532+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:21.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:20.827+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:21.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:20.827+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:21.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:20.827+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:21.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:20.828+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:21.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:21.018+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:21.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:21.018+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:21.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:21.018+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:21.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:21.019+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:21.809 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:21.530+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:21.809 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:21.530+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:21.809 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:21.530+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:21.809 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:21.531+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:22.273 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:22.013+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:22.273 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:22.013+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:22.273 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:22.013+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:22.273 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:22.014+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:22.274 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:21.809+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:22.274 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:21.809+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:22.274 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:21.809+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:22.274 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:21.810+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:22.987 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:22.568+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:22.987 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:22.568+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:22.987 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:22.568+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:22.987 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:22.569+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:22.987 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:22.787+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:22.988 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:22.787+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:22.988 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:22.787+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:22.988 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:22.788+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:23.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:22.987+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:23.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:22.987+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:23.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:22.987+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:23.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:22.988+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:23.837 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:23.582+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:23.837 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:23.582+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:23.837 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:23.582+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:23.837 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:23.583+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:24.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:23.837+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:24.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:23.837+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:24.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:23.837+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:24.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:23.838+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:24.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:23.989+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:24.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:23.989+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:24.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:23.989+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:24.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:23.990+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:24.990 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:24.619+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:24.990 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:24.619+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:24.990 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:24.619+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:24.990 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:24.620+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:24.991 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:24.824+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:24.991 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:24.824+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:24.991 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:24.824+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:24.991 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:24.825+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:25.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:35:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:35:24.990+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 29996 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:35:25.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:25.010+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:25.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:25.010+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:25.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:25.010+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:25.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:25.011+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:26.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:25.664+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:26.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:25.664+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:26.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:25.664+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:26.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:25.665+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:26.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:25.872+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:26.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:25.872+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:26.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:25.872+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:26.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:25.874+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:26.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:26.039+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:26.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:26.039+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:26.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:26.039+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:26.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:26.040+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:27.019 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:26.652+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:27.020 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:26.652+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:27.020 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:26.652+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:27.020 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:26.653+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:27.020 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:26.831+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:27.020 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:26.831+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:27.020 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:26.831+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:27.020 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:26.832+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:27.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:27.020+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:27.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:27.020+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:27.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:27.020+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:27.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:27.021+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:28.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:27.845+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:28.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:27.845+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:28.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:27.846+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:28.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:27.846+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:28.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:27.681+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:28.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:27.681+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:28.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:27.681+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:28.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:27.682+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:28.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:28.061+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:28.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:28.061+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:28.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:28.061+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:28.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:28.061+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:29.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:28.827+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:29.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:28.827+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:29.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:28.827+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:29.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:28.828+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:29.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:28.658+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:29.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:28.658+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:29.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:28.658+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:29.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:28.659+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:29.383 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:29.087+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:29.383 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:29.087+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:29.383 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:29.087+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:29.383 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:29.087+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:29.994 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:29.678+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:29.994 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:29.678+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:29.994 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:29.678+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:29.994 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:29.678+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:29.994 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:29.873+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:29.995 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:29.873+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:29.995 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:29.873+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:29.995 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:29.873+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:30.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:35:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:35:29.994+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 30079 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:35:30.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:30.055+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:30.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:30.055+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:30.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:30.055+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:30.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:30.056+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:31.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:30.902+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:31.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:30.902+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:31.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:30.902+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:31.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:30.903+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:31.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:30.706+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:31.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:30.706+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:31.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:30.706+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:31.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:30.707+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:31.490 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:31.084+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:31.490 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:31.084+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:31.491 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:31.084+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:31.491 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:31.085+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:31.753 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:31.673+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:31.753 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:31.673+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:31.753 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:31.673+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:31.754 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:31.674+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:32.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:31.934+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:32.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:31.934+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:32.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:31.934+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:32.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:31.935+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:32.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:32.073+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:32.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:32.073+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:32.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:32.073+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:32.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:32.074+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:32.935 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:32.653+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:32.935 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:32.653+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:32.935 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:32.653+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:32.935 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:32.654+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:33.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:32.936+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:33.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:32.936+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:33.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:32.936+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:33.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:32.937+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:33.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:33.052+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:33.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:33.052+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:33.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:33.052+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:33.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:33.053+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:33.922 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:33.631+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:33.922 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:33.631+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:33.922 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:33.631+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:33.923 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:33.632+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:34.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:33.923+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:34.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:33.923+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:34.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:33.923+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:34.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:33.924+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:34.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:34.032+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:34.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:34.032+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:34.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:34.032+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:34.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:34.033+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:34.878 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:34.647+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:34.878 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:34.647+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:34.878 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:34.647+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:34.878 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:34.648+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:35.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:35:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:35:34.998+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 30163 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:35:35.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:34.878+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:35.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:34.878+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:35.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:34.878+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:35.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:34.879+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:35.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:35.051+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:35.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:35.051+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:35.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:35.051+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:35.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:35.052+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:36.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:35.626+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:36.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:35.626+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:36.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:35.626+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:36.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:35.626+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:36.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:35.851+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:36.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:35.851+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:36.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:35.851+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:36.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:35.852+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:36.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:36.085+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:36.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:36.085+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:36.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:36.085+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:36.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:36.086+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:36.884 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:36.628+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:36.884 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:36.628+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:36.884 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:36.628+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:36.884 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:36.629+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:37.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:37.071+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:37.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:37.071+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:37.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:37.071+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:37.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:37.072+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:37.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:36.885+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:37.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:36.885+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:37.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:36.885+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:37.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:36.886+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:38.023 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:37.876+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:38.023 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:37.876+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:38.023 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:37.876+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:38.023 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:37.877+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:38.024 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:37.658+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:38.024 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:37.659+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:38.024 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:37.659+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:38.024 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:37.660+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:38.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:38.024+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:38.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:38.024+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:38.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:38.024+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:38.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:38.025+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:38.889 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:38.612+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:38.889 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:38.612+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:38.889 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:38.612+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:38.890 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:38.613+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:39.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:39.059+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:39.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:39.059+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:39.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:39.059+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:39.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:39.060+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:39.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:38.890+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:39.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:38.890+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:39.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:38.890+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:39.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:38.891+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:39.923 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:39.624+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:39.924 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:39.624+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:39.924 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:39.624+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:39.924 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:39.625+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:40.184 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:35:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:35:40.001+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 30246 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:35:40.184 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:40.085+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:40.185 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:40.085+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:40.185 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:40.085+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:40.185 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:40.086+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:40.185 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:39.924+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:40.185 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:39.924+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:40.185 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:39.924+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:40.185 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:39.925+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:40.973 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:40.665+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:40.973 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:40.665+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:40.974 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:40.665+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:40.974 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:40.666+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:41.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:40.974+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:41.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:40.974+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:41.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:40.974+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:41.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:40.975+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:41.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:41.116+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:41.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:41.116+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:41.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:41.116+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:41.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:41.116+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:41.989 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:41.705+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:41.989 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:41.705+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:41.990 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:41.705+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:41.990 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:41.706+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:42.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:41.990+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:42.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:41.990+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:42.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:41.990+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:42.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:41.991+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:42.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:42.068+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:42.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:42.068+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:42.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:42.068+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:42.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:42.068+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:43.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:42.656+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:43.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:42.656+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:43.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:42.656+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:43.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:42.657+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:43.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:43.033+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:43.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:43.033+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:43.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:43.033+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:43.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:43.033+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:43.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:43.103+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:43.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:43.103+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:43.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:43.103+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:43.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:43.104+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:44.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:43.664+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:44.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:43.664+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:44.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:43.664+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:44.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:43.665+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:44.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:44.039+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:44.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:44.039+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:44.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:44.039+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:44.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:44.040+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:44.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:44.148+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:44.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:44.148+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:44.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:44.148+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:44.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:44.149+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:45.005 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:44.687+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:45.005 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:44.687+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:45.005 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:44.687+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:45.005 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:44.688+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:45.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:35:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:35:45.005+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 30330 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:35:45.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:45.199+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:45.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:45.199+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:45.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:45.199+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:45.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:45.200+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:45.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:45.064+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:45.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:45.064+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:45.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:45.064+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:45.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:45.065+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:46.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:45.689+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:46.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:45.689+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:46.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:45.689+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:46.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:45.690+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:46.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:46.192+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:46.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:46.192+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:46.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:46.192+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:46.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:46.192+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:46.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:46.097+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:46.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:46.097+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:46.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:46.097+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:46.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:46.098+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:47.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:46.679+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:47.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:46.679+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:47.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:46.679+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:47.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:46.680+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:47.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:47.061+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:47.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:47.061+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:47.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:47.061+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:47.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:47.061+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:47.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:47.234+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:47.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:47.234+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:47.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:47.234+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:47.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:47.236+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:48.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:47.698+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:48.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:47.698+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:48.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:47.698+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:48.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:47.699+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:48.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:48.210+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:48.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:48.210+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:48.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:48.210+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:48.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:48.211+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:48.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:48.032+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:48.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:48.032+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:48.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:48.032+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:48.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:48.033+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:49.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:48.738+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:49.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:48.738+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:49.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:48.738+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:49.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:48.740+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:49.383 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:49.175+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:49.383 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:49.175+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:49.383 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:49.175+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:49.384 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:49.176+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:49.384 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:49.043+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:49.384 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:49.043+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:49.384 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:49.043+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:49.384 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:49.044+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:50.008 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:49.700+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:50.008 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:49.700+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:50.008 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:49.700+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:50.008 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:49.701+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:50.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:35:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:35:50.008+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 30413 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:35:50.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:50.160+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:50.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:50.160+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:50.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:50.160+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:50.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:50.160+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:50.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:50.013+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:50.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:50.013+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:50.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:50.013+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:50.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:50.014+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:51.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:50.751+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:51.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:50.751+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:51.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:50.751+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:51.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:50.752+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:51.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:51.038+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:51.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:51.038+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:51.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:51.038+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:51.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:51.039+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:51.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:51.122+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:51.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:51.122+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:51.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:51.122+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:51.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:51.124+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:52.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:51.704+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:52.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:51.704+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:52.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:51.704+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:52.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:51.705+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:52.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:52.082+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:52.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:52.082+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:52.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:52.082+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:52.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:52.082+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:52.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:52.127+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:52.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:52.127+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:52.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:52.127+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:52.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:52.128+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:52.993 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:52.732+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:52.993 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:52.732+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:52.993 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:52.732+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:52.993 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:52.734+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:53.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:53.089+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:53.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:53.089+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:53.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:53.089+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:53.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:53.090+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:53.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:53.152+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:53.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:53.152+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:53.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:53.152+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:53.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:53.153+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:54.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:53.748+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:54.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:53.748+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:54.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:53.748+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:54.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:53.748+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:54.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:54.141+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:54.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:54.141+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:54.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:54.141+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:54.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:54.142+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:54.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:54.139+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:54.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:54.139+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:54.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:54.139+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:54.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:54.140+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:55.011 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:54.744+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:55.011 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:54.744+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:55.011 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:54.744+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:55.012 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:54.745+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:55.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:35:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:35:55.012+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 30497 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:35:55.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:55.187+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:55.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:55.187+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:55.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:55.187+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:55.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:55.188+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:55.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:55.141+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:55.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:55.141+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:55.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:55.141+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:55.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:55.141+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:56.065 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:55.791+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:56.066 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:55.791+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:56.066 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:55.791+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:56.066 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:55.792+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:56.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:56.158+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:56.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:56.158+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:56.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:56.158+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:56.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:56.159+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:56.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:56.150+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:56.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:56.150+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:56.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:56.150+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:56.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:56.151+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:57.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:56.749+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:57.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:56.749+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:57.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:56.749+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:57.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:56.750+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:57.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:57.170+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:57.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:57.170+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:57.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:57.170+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:57.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:57.171+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:57.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:57.191+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:57.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:57.191+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:57.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:57.192+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:57.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:57.192+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:58.202 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:57.785+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:58.202 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:57.785+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:58.202 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:57.785+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:58.202 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:57.785+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:58.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:58.203+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:58.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:58.203+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:58.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:58.203+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:58.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:58.204+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:58.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:58.239+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:58.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:58.239+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:58.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:58.239+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:58.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:58.240+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:59.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:58.741+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:59.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:58.741+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:59.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:58.741+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:35:59.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:58.742+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:59.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:59.156+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:59.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:59.156+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:59.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:59.156+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:35:59.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:35:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:35:59.157+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:35:59.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:59.254+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:59.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:59.254+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:59.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:59.254+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:35:59.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:35:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:35:59.255+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:00.014 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:59.710+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:00.015 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:59.710+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:00.015 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:59.710+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:00.015 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:35:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:35:59.711+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:00.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:36:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:36:00.015+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 30577 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:36:00.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:00.155+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:00.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:00.155+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:00.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:00.155+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:00.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:00.156+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:00.696 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:00.291+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:00.696 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:00.291+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:00.696 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:00.291+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:00.696 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:00.292+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:01.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:00.696+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:01.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:00.696+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:01.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:00.696+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:01.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:00.697+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:01.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:01.172+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:01.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:01.172+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:01.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:01.172+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:01.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:01.173+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:01.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:01.337+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:01.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:01.337+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:01.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:01.337+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:01.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:01.338+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:02.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:01.722+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:02.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:01.722+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:02.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:01.722+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:02.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:01.723+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:02.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:02.302+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:02.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:02.302+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:02.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:02.302+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:02.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:02.303+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:02.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:02.140+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:02.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:02.140+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:02.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:02.140+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:02.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:02.141+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:03.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:02.691+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:03.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:02.692+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:03.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:02.692+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:03.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:02.693+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:03.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:03.312+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:03.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:03.312+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:03.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:03.312+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:03.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:03.313+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:03.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:03.092+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:03.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:03.092+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:03.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:03.092+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:03.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:03.093+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:04.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:03.654+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:04.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:03.654+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:04.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:03.654+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:04.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:03.655+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:04.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:04.326+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:04.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:04.326+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:04.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:04.326+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:04.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:04.327+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:04.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:04.094+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:04.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:04.094+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:04.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:04.094+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:04.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:04.095+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:05.018 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:04.626+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:05.018 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:04.626+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:05.019 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:04.626+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:05.019 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:04.627+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:05.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:36:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:36:05.019+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 30658 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:36:05.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:05.105+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:05.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:05.105+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:05.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:05.105+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:05.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:05.105+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:05.615 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:05.284+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:05.615 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:05.284+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:05.615 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:05.284+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:05.615 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:05.285+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:06.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:05.616+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:06.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:05.616+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:06.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:05.616+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:06.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:05.617+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:06.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:06.113+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:06.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:06.113+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:06.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:06.113+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:06.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:06.114+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:06.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:06.304+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:06.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:06.304+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:06.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:06.304+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:06.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:06.305+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:07.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:06.625+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:07.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:06.625+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:07.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:06.625+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:07.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:06.626+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:07.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:07.150+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:07.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:07.150+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:07.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:07.150+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:07.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:07.151+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:07.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:07.312+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:07.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:07.312+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:07.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:07.312+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:07.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:07.313+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:08.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:07.602+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:08.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:07.602+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:08.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:07.602+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:08.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:07.603+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:08.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:08.121+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:08.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:08.121+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:08.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:08.121+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:08.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:08.122+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:08.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:08.282+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:08.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:08.282+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:08.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:08.282+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:08.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:08.283+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:09.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:08.591+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:09.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:08.591+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:09.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:08.591+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:09.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:08.592+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:09.383 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:09.172+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:09.383 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:09.172+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:09.383 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:09.172+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:09.383 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:09.173+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:09.384 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:09.246+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:09.384 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:09.246+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:09.384 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:09.246+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:09.384 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:09.247+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:10.022 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:09.635+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:10.022 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:09.635+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:10.022 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:09.635+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:10.023 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:09.636+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:10.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:36:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:36:10.023+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 30735 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:36:10.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:10.202+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:10.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:10.202+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:10.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:10.202+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:10.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:10.202+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:10.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:10.210+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:10.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:10.210+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:10.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:10.210+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:10.276 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:10.211+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:11.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:10.630+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:11.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:10.630+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:11.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:10.630+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:11.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:10.631+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:11.425 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:11.223+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:11.425 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:11.223+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:11.425 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:11.223+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:11.425 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:11.224+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:11.426 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:11.176+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:11.426 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:11.176+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:11.426 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:11.176+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:11.426 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:11.176+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:11.756 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:11.601+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:11.756 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:11.601+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:11.757 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:11.601+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:11.757 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:11.602+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:12.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:12.204+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:12.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:12.204+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:12.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:12.204+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:12.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:12.204+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:12.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:12.207+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:12.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:12.207+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:12.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:12.207+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:12.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:12.208+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:13.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:12.619+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:13.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:12.619+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:13.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:12.619+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:13.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:12.620+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:13.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:13.251+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:13.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:13.251+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:13.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:13.251+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:13.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:13.252+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:13.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:13.190+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:13.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:13.190+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:13.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:13.190+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:13.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:13.190+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:14.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:13.666+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:14.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:13.666+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:14.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:13.666+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:14.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:13.667+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:14.497 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:14.166+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:14.497 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:14.166+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:14.498 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:14.166+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:14.498 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:14.166+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:14.498 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:14.204+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:14.498 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:14.204+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:14.498 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:14.204+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:14.498 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:14.204+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:14.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:14.707+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:14.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:14.707+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:14.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:14.707+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:14.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:14.708+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:15.525 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:36:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:36:15.026+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 30810 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:36:15.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:15.197+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:15.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:15.197+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:15.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:15.197+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:15.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:15.197+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:15.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:15.179+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:15.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:15.179+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:15.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:15.179+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:15.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:15.179+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:16.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:15.711+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:16.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:15.711+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:16.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:15.711+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:16.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:15.712+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:16.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:16.211+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:16.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:16.211+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:16.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:16.211+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:16.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:16.212+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:16.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:16.205+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:16.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:16.205+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:16.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:16.205+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:16.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:16.206+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:17.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:16.751+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:17.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:16.751+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:17.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:16.751+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:17.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:16.752+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:17.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:17.233+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:17.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:17.233+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:17.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:17.233+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:17.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:17.234+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:17.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:17.239+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:17.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:17.239+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:17.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:17.239+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:17.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:17.240+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:18.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:17.740+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:18.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:17.740+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:18.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:17.740+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:18.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:17.741+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:18.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:18.197+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:18.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:18.197+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:18.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:18.197+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:18.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:18.198+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:18.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:18.214+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:18.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:18.214+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:18.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:18.214+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:18.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:18.214+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:19.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:18.701+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:19.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:18.701+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:19.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:18.701+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:19.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:18.701+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:19.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:19.200+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:19.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:19.200+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:19.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:19.200+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:19.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:19.201+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:19.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:19.181+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:19.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:19.181+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:19.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:19.181+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:19.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:19.182+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:20.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:19.703+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:20.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:19.703+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:20.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:19.703+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:20.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:19.704+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:20.525 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:36:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:36:20.030+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 30887 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:36:20.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:20.206+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:20.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:20.206+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:20.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:20.206+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:20.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:20.208+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:20.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:20.169+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:20.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:20.169+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:20.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:20.169+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:20.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:20.170+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:21.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:20.737+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:21.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:20.737+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:21.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:20.737+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:21.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:20.738+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:21.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:21.207+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:21.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:21.207+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:21.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:21.207+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:21.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:21.208+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:21.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:21.199+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:21.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:21.199+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:21.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:21.199+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:21.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:21.200+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:22.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:21.779+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:22.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:21.779+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:22.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:21.779+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:22.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:21.780+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:22.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:22.214+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:22.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:22.214+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:22.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:22.214+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:22.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:22.215+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:22.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:22.198+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:22.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:22.198+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:22.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:22.198+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:22.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:22.199+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:23.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:22.752+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:23.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:22.752+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:23.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:22.752+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:23.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:22.753+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:23.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:23.254+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:23.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:23.254+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:23.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:23.254+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:23.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:23.255+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:23.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:23.178+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:23.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:23.178+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:23.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:23.178+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:23.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:23.179+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:24.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:23.780+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:24.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:23.780+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:24.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:23.780+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:24.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:23.781+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:24.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:24.131+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:24.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:24.131+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:24.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:24.131+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:24.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:24.131+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:24.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:24.211+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:24.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:24.211+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:24.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:24.211+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:24.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:24.212+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:25.111 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:36:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:36:25.034+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 30965 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:36:25.111 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:24.816+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:25.111 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:24.816+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:25.112 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:24.816+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:25.112 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:24.817+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:25.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:25.262+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:25.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:25.262+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:25.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:25.262+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:25.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:25.263+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:25.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:25.112+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:25.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:25.112+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:25.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:25.112+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:25.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:25.113+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:26.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:25.770+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:26.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:25.770+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:26.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:25.770+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:26.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:25.771+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:26.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:26.233+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:26.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:26.233+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:26.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:26.233+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:26.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:26.234+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:26.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:26.119+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:26.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:26.119+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:26.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:26.120+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:26.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:26.120+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:27.124 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:26.804+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:27.124 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:26.804+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:27.124 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:26.804+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:27.125 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:26.805+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:27.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:27.125+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:27.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:27.125+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:27.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:27.125+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:27.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:27.126+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:27.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:27.257+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:27.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:27.257+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:27.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:27.257+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:27.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:27.258+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:28.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:27.765+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:28.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:27.765+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:28.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:27.765+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:28.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:27.766+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:28.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:28.261+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:28.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:28.261+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:28.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:28.261+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:28.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:28.261+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:28.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:28.119+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:28.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:28.119+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:28.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:28.119+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:28.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:28.119+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:29.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:28.738+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:29.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:28.738+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:29.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:28.738+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:29.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:28.739+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:29.383 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:29.268+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:29.383 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:29.268+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:29.383 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:29.268+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:29.384 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:29.269+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:29.384 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:29.118+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:29.384 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:29.118+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:29.384 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:29.118+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:29.384 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:29.119+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:30.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:29.723+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:30.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:29.723+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:30.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:29.723+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:30.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:29.724+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:30.525 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:36:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:36:30.038+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 31048 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:36:30.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:30.260+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:30.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:30.260+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:30.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:30.260+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:30.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:30.261+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:30.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:30.109+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:30.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:30.109+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:30.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:30.109+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:30.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:30.110+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:31.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:30.744+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:31.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:30.744+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:31.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:30.744+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:31.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:30.745+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:31.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:31.249+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:31.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:31.249+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:31.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:31.249+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:31.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:31.249+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:31.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:31.149+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:31.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:31.149+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:31.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:31.149+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:31.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:31.149+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:32.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:31.757+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:32.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:31.757+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:32.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:31.757+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:32.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:31.758+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:32.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:32.252+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:32.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:32.252+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:32.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:32.252+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:32.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:32.253+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:32.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:32.198+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:32.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:32.198+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:32.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:32.198+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:32.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:32.199+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:33.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:32.752+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:33.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:32.752+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:33.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:32.752+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:33.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:32.753+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:33.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:33.227+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:33.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:33.227+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:33.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:33.227+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:33.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:33.227+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:33.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:33.202+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:33.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:33.202+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:33.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:33.202+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:33.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:33.203+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:34.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:33.790+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:34.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:33.790+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:34.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:33.790+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:34.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:33.791+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:34.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:34.232+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:34.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:34.232+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:34.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:34.232+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:34.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:34.233+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:34.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:34.214+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:34.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:34.214+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:34.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:34.214+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:34.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:34.215+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:35.049 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:36:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:36:35.042+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 31130 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:36:35.049 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:34.794+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:35.049 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:34.794+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:35.049 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:34.794+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:35.050 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:34.795+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:35.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:35.280+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:35.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:35.280+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:35.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:35.280+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:35.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:35.281+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:35.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:35.225+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:35.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:35.225+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:35.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:35.225+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:35.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:35.225+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:36.253 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:35.790+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:36.253 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:35.790+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:36.253 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:35.790+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:36.253 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:35.791+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:36.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:36.321+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:36.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:36.321+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:36.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:36.321+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:36.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:36.322+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:36.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:36.253+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:36.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:36.253+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:36.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:36.253+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:36.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:36.255+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:37.232 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:36.784+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:37.232 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:36.784+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:37.232 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:36.784+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:37.232 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:36.785+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:37.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:37.282+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:37.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:37.283+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:37.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:37.283+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:37.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:37.283+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:37.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:37.232+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:37.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:37.232+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:37.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:37.232+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:37.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:37.233+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:38.241 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:37.815+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:38.242 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:37.815+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:38.242 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:37.815+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:38.242 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:37.816+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:38.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:38.303+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:38.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:38.303+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:38.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:38.303+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:38.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:38.304+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:38.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:38.242+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:38.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:38.242+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:38.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:38.242+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:38.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:38.243+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:39.271 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:38.792+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:39.271 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:38.792+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:39.272 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:38.792+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:39.272 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:38.793+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:39.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:39.338+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:39.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:39.338+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:39.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:39.338+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:39.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:39.338+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:39.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:39.272+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:39.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:39.272+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:39.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:39.272+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:39.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:39.273+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:40.045 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:39.790+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:40.046 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:39.790+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:40.046 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:39.790+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:40.046 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:39.791+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:40.369 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:36:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:36:40.046+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 31213 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:36:40.369 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:40.284+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:40.369 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:40.284+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:40.369 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:40.284+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:40.369 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:40.284+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:40.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:40.370+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:40.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:40.370+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:40.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:40.370+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:40.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:40.371+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:41.236 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:40.813+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:41.236 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:40.813+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:41.236 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:40.813+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:41.236 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:40.814+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:41.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:41.370+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:41.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:41.370+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:41.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:41.370+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:41.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:41.370+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:41.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:41.237+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:41.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:41.237+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:41.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:41.237+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:41.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:41.238+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:42.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:41.822+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:42.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:41.822+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:42.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:41.822+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:42.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:41.823+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:42.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:42.326+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:42.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:42.326+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:42.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:42.326+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:42.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:42.327+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:42.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:42.201+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:42.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:42.201+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:42.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:42.201+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:42.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:42.202+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:43.192 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:42.801+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:43.192 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:42.801+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:43.192 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:42.801+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:43.192 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:42.802+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:43.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:43.346+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:43.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:43.346+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:43.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:43.346+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:43.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:43.347+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:43.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:43.192+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:43.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:43.192+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:43.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:43.192+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:43.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:43.193+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:44.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:43.774+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:44.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:43.774+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:44.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:43.774+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:44.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:43.775+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:44.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:44.389+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:44.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:44.389+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:44.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:44.389+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:44.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:44.390+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:44.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:44.155+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:44.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:44.155+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:44.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:44.155+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:44.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:44.156+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:45.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:44.756+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:45.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:44.756+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:45.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:44.756+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:45.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:44.757+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:45.358 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:36:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:36:45.050+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 31297 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:36:45.359 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:45.186+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:45.359 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:45.186+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:45.359 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:45.186+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:45.359 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:45.187+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:45.736 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:45.359+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:45.736 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:45.359+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:45.737 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:45.359+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:45.737 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:45.360+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:46.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:45.737+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:46.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:45.737+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:46.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:45.737+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:46.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:45.738+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:46.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:46.214+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:46.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:46.214+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:46.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:46.214+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:46.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:46.215+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:46.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:46.356+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:46.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:46.356+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:46.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:46.356+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:46.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:46.357+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:47.235 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:46.778+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:47.235 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:46.778+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:47.235 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:46.778+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:47.235 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:46.779+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:47.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:47.236+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:47.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:47.236+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:47.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:47.236+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:47.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:47.237+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:47.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:47.392+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:47.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:47.392+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:47.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:47.392+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:47.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:47.393+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:48.191 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:47.810+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:48.191 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:47.810+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:48.191 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:47.810+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:48.191 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:47.811+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:48.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:48.427+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:48.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:48.427+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:48.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:48.427+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:48.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:48.428+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:48.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:48.191+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:48.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:48.191+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:48.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:48.191+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:48.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:48.192+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:49.199 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:48.829+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:49.199 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:48.829+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:49.199 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:48.829+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:49.199 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:48.830+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:49.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:49.200+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:49.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:49.200+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:49.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:49.200+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:49.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:49.201+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:49.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:49.411+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:49.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:49.411+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:49.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:49.411+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:49.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:49.412+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:50.176 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:36:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:36:50.054+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 31380 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:36:50.177 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:49.842+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:50.177 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:49.842+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:50.177 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:49.842+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:50.177 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:49.843+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:50.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:50.387+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:50.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:50.387+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:50.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:50.387+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:50.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:50.388+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:50.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:50.199+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:50.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:50.199+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:50.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:50.199+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:50.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:50.200+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:51.156 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:50.850+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:51.157 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:50.850+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:51.157 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:50.850+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:51.157 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:50.851+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:51.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:51.381+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:51.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:51.382+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:51.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:51.382+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:51.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:51.383+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:51.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:51.157+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:51.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:51.157+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:51.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:51.157+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:51.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:51.158+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:52.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:51.877+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:52.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:51.877+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:52.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:51.877+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:52.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:51.878+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:52.402 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:52.118+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:52.403 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:52.118+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:52.403 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:52.118+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:52.403 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:52.119+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:52.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:52.403+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:52.848 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:52.403+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:52.848 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:52.403+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:52.848 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:52.404+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:53.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:53.119+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:53.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:53.119+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:53.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:53.119+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:53.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:53.120+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:53.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:52.925+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:53.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:52.925+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:53.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:52.925+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:53.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:52.925+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:53.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:53.406+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:53.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:53.406+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:53.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:53.406+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:53.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:53.407+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:54.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:53.969+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:54.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:53.969+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:54.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:53.969+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:54.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:53.970+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:54.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:54.157+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:54.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:54.157+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:54.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:54.157+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:54.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:54.157+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:54.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:54.425+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:54.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:54.425+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:54.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:54.425+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:54.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:54.426+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:55.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:36:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:36:55.058+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 31464 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:36:55.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:55.153+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:55.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:55.153+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:55.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:55.153+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:55.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:55.153+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:55.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:54.989+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:55.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:54.989+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:55.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:54.989+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:55.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:54.991+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:55.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:55.389+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:55.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:55.389+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:55.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:55.389+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:55.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:55.389+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:56.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:56.138+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:56.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:56.138+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:56.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:56.138+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:56.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:56.138+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:56.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:56.025+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:56.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:56.025+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:56.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:56.025+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:56.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:56.026+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:56.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:56.387+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:56.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:56.387+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:56.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:56.387+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:56.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:56.388+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:57.390 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:57.185+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:57.390 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:57.185+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:57.391 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:57.185+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:57.391 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:57.185+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:57.391 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:57.048+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:57.391 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:57.048+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:57.391 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:57.048+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:57.391 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:57.049+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:57.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:57.391+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:57.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:57.391+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:57.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:57.391+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:57.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:57.392+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:58.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:58.024+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:58.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:58.024+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:58.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:58.024+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:58.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:58.025+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:58.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:58.169+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:58.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:58.169+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:58.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:58.169+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:58.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:58.170+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:58.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:58.438+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:58.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:58.438+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:58.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:58.438+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:58.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:58.439+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:59.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:59.014+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:59.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:59.014+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:59.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:59.014+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:36:59.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:36:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:36:59.015+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:59.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:59.128+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:59.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:59.128+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:59.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:59.128+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:36:59.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:36:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:36:59.129+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:36:59.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:59.398+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:59.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:59.398+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:59.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:59.398+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:36:59.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:36:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:36:59.399+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:00.275 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:37:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:37:00.062+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 31546 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:37:00.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:00.007+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:00.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:00.007+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:00.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:00.007+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:00.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:00.007+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:00.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:00.158+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:00.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:00.158+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:00.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:00.158+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:00.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:00.159+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:00.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:00.407+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:00.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:00.407+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:00.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:00.407+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:00.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:00.408+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:01.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:01.019+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:01.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:01.019+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:01.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:01.019+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:01.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:01.020+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:01.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:01.151+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:01.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:01.151+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:01.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:01.151+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:01.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:01.152+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:01.759 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:01.426+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:01.760 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:01.426+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:01.760 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:01.426+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:01.760 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:01.427+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:02.474 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:02.176+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:02.474 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:02.176+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:02.475 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:02.176+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:02.475 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:02.177+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:02.475 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:02.028+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:02.475 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:02.028+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:02.475 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:02.028+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:02.475 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:02.029+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:02.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:02.475+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:02.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:02.475+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:02.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:02.475+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:02.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:02.476+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:03.464 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:03.064+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:03.465 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:03.064+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:03.465 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:03.064+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:03.465 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:03.065+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:03.465 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:03.217+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:03.465 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:03.217+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:03.465 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:03.217+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:03.466 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:03.218+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:03.774 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:03.465+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:03.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:03.465+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:03.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:03.465+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:03.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:03.466+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:04.498 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:04.224+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:04.498 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:04.224+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:04.498 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:04.224+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:04.498 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:04.225+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:04.499 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:04.104+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:04.499 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:04.104+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:04.499 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:04.104+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:04.499 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:04.104+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:04.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:04.499+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:04.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:04.499+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:04.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:04.499+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:04.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:04.499+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:05.485 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:37:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:37:05.066+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 31631 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:37:05.486 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:05.144+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:05.486 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:05.144+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:05.486 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:05.144+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:05.486 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:05.145+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:05.486 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:05.186+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:05.486 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:05.186+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:05.487 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:05.186+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:05.487 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:05.187+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:05.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:05.486+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:05.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:05.486+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:05.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:05.486+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:05.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:05.487+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:06.046 ERROR:paramiko.transport:Socket exception: No route to host (113) 2024-03-11T23:37:06.051 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T23:37:06.051 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-03-11T23:37:06.052 ERROR:teuthology.run_tasks:Manager failed: kclient Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-03-11T23:37:06.065 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-11T23:37:06.076 INFO:tasks.cephadm:Teardown begin 2024-03-11T23:37:06.076 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/cephadm.py", line 1807, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-03-11T23:37:06.081 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T23:37:06.115 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:37:06.115 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:37:06.518 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:06.177+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:06.519 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:06.177+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:06.519 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:06.177+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:06.519 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:06.178+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:06.519 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:06.103+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:06.519 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:06.103+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:06.519 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:06.103+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:06.519 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:06.105+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:06.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:06.519+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:06.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:06.519+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:06.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:06.519+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:06.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:06.520+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:07.500 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:07.223+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:07.501 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:07.223+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:07.501 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:07.223+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:07.501 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:07.223+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:07.501 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:07.087+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:07.501 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:07.087+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:07.501 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:07.087+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:07.502 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:07.087+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:07.774 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:07.501+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:07.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:07.501+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:07.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:07.501+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:07.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:07.503+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:08.499 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:08.222+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:08.499 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:08.222+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:08.499 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:08.222+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:08.499 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:08.222+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:08.499 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:08.053+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:08.500 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:08.053+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:08.500 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:08.053+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:08.500 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:08.054+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:08.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:08.499+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:08.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:08.499+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:08.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:08.499+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:08.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:08.500+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:09.182 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:37:09.382 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:09.198+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:09.383 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:09.198+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:09.383 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:09.198+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:09.383 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:09.199+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:09.383 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:09.064+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:09.383 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:09.064+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:09.383 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:09.064+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:09.384 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:09.064+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:09.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:09.450+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:09.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:09.450+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:09.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:09.450+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:09.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:09.451+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:10.450 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:37:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:37:10.070+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 31713 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:37:10.451 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:10.037+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:10.451 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:10.037+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:10.451 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:10.037+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:10.451 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:10.038+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:10.452 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:10.235+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:10.452 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:10.235+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:10.452 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:10.235+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:10.452 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:10.236+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:10.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:10.451+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:10.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:10.451+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:10.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:10.451+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:10.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:10.452+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:11.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:11.205+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:11.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:11.205+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:11.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:11.205+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:11.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:11.206+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:11.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:11.019+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:11.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:11.019+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:11.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:11.019+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:11.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:11.020+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:11.760 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:11.451+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:11.760 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:11.451+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:11.760 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:11.451+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:11.760 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:11.452+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:12.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:11.975+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:12.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:11.975+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:12.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:11.975+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:12.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:11.977+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:12.444 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:12.168+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:12.444 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:12.168+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:12.444 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:12.168+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:12.444 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:12.168+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:12.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:12.444+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:12.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:12.444+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:12.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:12.444+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:12.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:12.446+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:13.140 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:13.024+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:13.140 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:13.024+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:13.140 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:13.024+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:13.140 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:13.024+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:13.478 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:13.142+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:13.478 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:13.142+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:13.478 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:13.142+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:13.478 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:13.142+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:13.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:13.478+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:13.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:13.478+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:13.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:13.478+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:13.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:13.479+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:14.443 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:14.125+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:14.444 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:14.125+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:14.444 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:14.125+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:14.444 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:14.126+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:14.444 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:14.056+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:14.445 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:14.056+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:14.445 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:14.056+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:14.445 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:14.057+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:14.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:14.444+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:14.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:14.444+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:14.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:14.444+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:14.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:14.445+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:15.430 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:37:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:37:15.074+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 31798 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:37:15.431 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:15.093+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:15.431 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:15.093+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:15.431 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:15.093+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:15.431 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:15.094+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:15.431 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:15.066+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:15.431 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:15.066+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:15.432 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:15.066+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:15.432 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:15.067+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:15.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:15.431+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:15.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:15.431+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:15.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:15.431+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:15.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:15.432+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:16.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:16.056+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:16.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:16.056+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:16.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:16.056+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:16.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:16.057+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:16.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:16.027+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:16.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:16.028+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:16.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:16.028+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:16.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:16.028+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:16.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:16.386+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:16.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:16.386+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:16.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:16.386+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:16.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:16.387+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:17.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:17.102+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:17.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:17.102+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:17.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:17.102+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:17.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:17.103+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:17.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:17.005+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:17.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:17.005+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:17.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:17.005+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:17.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:17.006+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:17.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:17.400+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:17.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:17.400+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:17.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:17.400+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:17.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:17.400+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:18.183 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:37:18.183 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:37:18.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:18.137+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:18.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:18.137+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:18.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:18.137+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:18.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:18.138+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:18.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:17.995+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:18.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:17.995+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:18.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:17.995+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:18.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:17.996+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:18.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:18.381+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:18.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:18.381+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:18.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:18.381+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:18.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:18.382+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:19.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:18.971+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:19.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:18.971+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:19.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:18.971+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:19.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:18.972+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:19.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:19.090+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:19.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:19.090+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:19.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:19.090+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:19.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:19.091+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:19.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:19.430+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:19.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:19.430+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:19.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:19.430+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:19.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:19.431+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:20.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:19.959+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:20.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:19.959+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:20.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:19.959+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:20.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:19.960+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:20.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:20.085+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:20.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:20.085+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:20.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:20.085+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:20.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:20.087+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:20.276 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:37:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:37:20.078+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 31880 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:37:20.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:20.387+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:20.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:20.387+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:20.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:20.387+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:20.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:20.388+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:21.246 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:37:21.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:21.071+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:21.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:21.071+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:21.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:21.071+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:21.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:21.072+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:21.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:20.931+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:21.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:20.931+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:21.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:20.931+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:21.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:20.932+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:21.761 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:21.419+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:21.761 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:21.419+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:21.761 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:21.419+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:21.761 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:21.420+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:22.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:21.979+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:22.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:21.979+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:22.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:21.979+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:22.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:21.980+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:22.445 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:22.110+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:22.445 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:22.111+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:22.446 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:22.111+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:22.446 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:22.111+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:22.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:22.446+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:22.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:22.446+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:22.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:22.446+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:22.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:22.447+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:23.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:23.154+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:23.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:23.154+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:23.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:23.154+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:23.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:23.155+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:23.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:23.013+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:23.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:23.013+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:23.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:23.013+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:23.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:23.014+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:23.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:23.438+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:23.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:23.438+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:23.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:23.438+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:23.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:23.439+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:24.462 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:24.175+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:24.462 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:24.175+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:24.462 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:24.175+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:24.462 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:24.176+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:24.462 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:24.026+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:24.463 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:24.026+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:24.463 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:24.026+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:24.463 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:24.027+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:24.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:24.462+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:24.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:24.462+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:24.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:24.462+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:24.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:24.463+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:25.493 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:37:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:37:25.082+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 31965 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:37:25.493 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:25.138+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:25.493 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:25.138+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:25.493 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:25.138+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:25.494 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:25.139+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:25.494 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:25.056+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:25.494 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:25.056+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:25.494 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:25.056+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:25.494 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:25.057+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:25.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:25.494+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:25.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:25.494+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:25.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:25.494+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:25.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:25.495+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:26.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:26.125+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:26.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:26.125+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:26.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:26.125+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:26.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:26.126+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:26.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:26.028+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:26.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:26.028+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:26.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:26.028+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:26.526 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:26.029+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:27.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:26.534+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:27.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:26.534+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:27.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:26.534+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:27.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:26.535+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:27.516 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:27.154+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:27.516 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:27.154+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:27.516 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:27.154+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:27.516 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:27.155+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:27.516 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:27.078+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:27.517 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:27.078+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:27.517 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:27.078+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:27.517 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:27.079+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:27.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:27.516+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:27.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:27.516+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:27.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:27.516+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:27.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:27.517+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:28.479 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:28.191+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:28.479 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:28.191+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:28.479 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:28.191+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:28.479 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:28.192+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:28.479 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:28.045+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:28.480 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:28.045+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:28.480 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:28.045+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:28.480 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:28.046+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:28.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:28.479+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:28.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:28.479+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:28.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:28.479+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:28.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:28.480+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:29.383 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:29.207+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:29.383 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:29.207+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:29.384 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:29.207+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:29.384 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:29.208+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:29.384 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:29.095+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:29.384 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:29.095+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:29.384 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:29.095+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:29.384 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:29.095+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:29.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:29.501+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:29.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:29.501+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:29.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:29.501+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:29.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:29.501+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:30.525 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:37:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:37:30.086+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 32045 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:37:30.525 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:30.080+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:30.525 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:30.080+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:30.525 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:30.080+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:30.525 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:30.080+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:30.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:30.190+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:30.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:30.190+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:30.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:30.190+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:30.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:30.191+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:31.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:30.540+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:31.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:30.540+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:31.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:30.540+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:31.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:30.540+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:31.522 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:31.102+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:31.523 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:31.102+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:31.523 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:31.102+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:31.523 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:31.103+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:31.523 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:31.144+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:31.524 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:31.144+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:31.524 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:31.144+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:31.524 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:31.145+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:31.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:31.523+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:31.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:31.523+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:31.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:31.523+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:31.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:31.524+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:32.516 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:32.136+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:32.516 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:32.136+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:32.516 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:32.136+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:32.516 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:32.136+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:32.517 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:32.130+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:32.517 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:32.130+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:32.517 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:32.130+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:32.517 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:32.131+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:32.774 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:32.517+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:32.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:32.517+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:32.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:32.517+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:32.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:32.518+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:33.247 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:37:33.247 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:37:33.502 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:33.088+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:33.502 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:33.088+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:33.502 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:33.088+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:33.502 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:33.088+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:33.502 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:33.118+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:33.503 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:33.118+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:33.503 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:33.118+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:33.503 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:33.119+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:33.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:33.502+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:33.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:33.502+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:33.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:33.502+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:33.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:33.504+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:34.369 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:34.100+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:34.369 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:34.100+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:34.369 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:34.100+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:34.369 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:34.100+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:34.370 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:34.135+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:34.370 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:34.135+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:34.370 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:34.135+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:34.370 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:34.136+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:34.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:34.541+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:34.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:34.541+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:34.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:34.541+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:34.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:34.542+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:35.519 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:37:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:37:35.090+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 32130 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:37:35.519 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:35.120+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:35.519 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:35.120+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:35.519 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:35.120+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:35.519 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:35.121+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:35.520 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:35.078+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:35.520 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:35.078+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:35.520 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:35.078+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:35.520 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:35.079+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:35.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:35.519+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:35.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:35.519+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:35.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:35.519+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:35.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:35.520+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:36.318 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:37:36.525 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:36.150+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:36.525 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:36.150+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:36.525 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:36.150+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:36.525 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:36.151+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:36.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:36.063+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:36.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:36.063+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:36.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:36.063+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:36.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:36.063+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:37.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:36.526+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:37.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:36.526+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:37.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:36.526+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:37.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:36.527+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:37.491 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:37.036+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:37.492 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:37.036+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:37.492 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:37.036+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:37.492 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:37.037+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:37.492 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:37.160+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:37.492 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:37.160+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:37.492 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:37.160+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:37.492 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:37.161+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:37.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:37.492+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:37.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:37.492+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:37.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:37.492+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:37.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:37.493+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:38.485 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:38.117+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:38.485 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:38.117+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:38.485 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:38.117+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:38.485 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:38.118+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:38.485 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:38.071+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:38.486 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:38.071+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:38.486 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:38.071+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:38.486 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:38.071+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:38.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:38.486+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:38.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:38.486+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:38.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:38.486+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:38.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:38.487+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:39.319 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:37:39.320 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:37:39.383 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:39.071+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:39.383 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:39.071+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:39.383 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:39.071+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:39.383 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:39.072+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:39.383 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:39.070+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:39.383 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:39.070+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:39.384 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:39.070+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:39.384 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:39.070+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:39.390 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:37:39.390 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-11T23:37:39.391 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-11T23:37:39.391 DEBUG:teuthology.orchestra.run.smithi153:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T23:37:39.405 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:37:39.406 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:37:39.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:39.474+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:39.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:39.474+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:39.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:39.474+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:39.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:39.475+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:40.457 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:37:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:37:40.094+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 32211 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:37:40.458 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:40.071+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:40.458 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:40.071+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:40.458 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:40.071+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:40.458 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:40.071+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:40.459 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:40.035+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:40.459 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:40.035+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:40.459 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:40.035+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:40.459 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:40.036+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:40.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:40.458+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:40.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:40.458+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:40.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:40.458+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:40.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:40.460+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:41.465 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:41.036+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:41.465 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:41.036+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:41.465 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:41.036+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:41.465 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:41.037+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:41.466 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:41.052+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:41.466 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:41.052+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:41.466 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:41.052+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:41.466 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:41.053+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:41.762 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:41.466+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:41.762 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:41.466+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:41.762 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:41.466+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:41.762 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:41.467+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:42.024 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:41.998+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:42.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:41.998+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:42.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:41.998+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:42.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:41.998+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:42.429 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:42.099+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:42.430 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:42.099+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:42.430 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:42.099+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:42.430 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:42.100+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:42.462 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:37:42.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:42.430+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:42.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:42.430+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:42.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:42.430+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:42.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:42.431+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:43.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:43.066+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:43.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:43.066+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:43.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:43.066+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:43.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:43.066+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:43.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:42.986+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:43.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:42.986+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:43.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:42.986+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:43.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:42.986+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:43.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:43.423+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:43.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:43.423+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:43.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:43.423+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:43.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:43.424+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:44.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:43.949+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:44.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:43.949+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:44.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:43.949+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:44.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:43.950+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:44.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:44.113+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:44.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:44.113+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:44.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:44.113+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:44.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:44.114+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:44.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:44.400+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:44.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:44.400+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:44.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:44.400+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:44.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:44.401+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:45.222 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:37:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:37:45.099+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 32296 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:37:45.222 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:44.917+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:45.222 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:44.917+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:45.222 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:44.917+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:45.222 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:44.917+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:45.223 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:45.077+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:45.223 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:45.077+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:45.223 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:45.077+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:45.223 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:45.078+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:45.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:45.428+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:45.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:45.428+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:45.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:45.428+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:45.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:45.429+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:46.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:46.095+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:46.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:46.095+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:46.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:46.095+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:46.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:46.096+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:46.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:45.896+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:46.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:45.896+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:46.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:45.896+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:46.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:45.897+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:46.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:46.425+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:46.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:46.425+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:46.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:46.425+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:46.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:46.426+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:47.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:47.096+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:47.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:47.096+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:47.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:47.096+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:47.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:47.097+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:47.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:46.928+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:47.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:46.928+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:47.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:46.928+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:47.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:46.929+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:47.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:47.389+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:47.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:47.389+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:47.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:47.389+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:47.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:47.389+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:48.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:48.089+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:48.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:48.089+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:48.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:48.089+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:48.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:48.090+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:48.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:47.924+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:48.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:47.924+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:48.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:47.924+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:48.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:47.925+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:48.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:48.389+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:48.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:48.389+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:48.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:48.389+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:48.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:48.390+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:49.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:49.132+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:49.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:49.132+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:49.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:49.132+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:49.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:49.133+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:49.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:48.934+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:49.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:48.934+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:49.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:48.934+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:49.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:48.934+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:49.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:49.380+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:49.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:49.380+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:49.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:49.380+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:49.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:49.381+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:50.222 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:37:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:37:50.103+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 32379 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:37:50.223 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:50.112+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:50.223 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:50.112+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:50.223 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:50.112+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:50.223 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:50.113+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:50.223 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:49.964+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:50.224 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:49.964+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:50.224 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:49.964+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:50.224 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:49.965+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:50.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:50.397+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:50.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:50.397+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:50.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:50.397+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:50.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:50.399+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:51.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:50.936+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:51.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:50.936+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:51.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:50.936+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:51.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:50.937+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:51.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:51.147+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:51.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:51.147+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:51.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:51.147+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:51.276 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:51.148+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:51.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:37:51.464 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:37:51.763 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:51.402+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:51.763 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:51.402+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:51.763 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:51.402+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:51.763 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:51.403+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:52.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:51.953+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:52.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:51.953+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:52.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:51.953+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:52.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:51.954+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:52.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:52.364+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:52.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:52.364+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:52.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:52.364+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:52.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:52.365+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:52.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:52.182+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:52.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:52.182+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:52.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:52.182+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:52.527 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:52.183+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:53.229 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:52.922+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:53.229 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:52.922+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:53.229 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:52.922+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:53.229 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:52.923+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:53.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:53.230+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:53.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:53.230+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:53.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:53.230+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:53.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:53.231+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:53.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:53.409+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:53.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:53.409+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:53.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:53.409+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:53.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:53.409+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:54.263 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:53.916+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:54.263 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:53.916+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:54.263 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:53.916+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:54.264 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:53.917+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:54.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:54.264+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:54.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:54.264+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:54.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:54.264+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:54.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:54.265+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:54.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:54.388+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:54.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:54.388+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:54.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:54.388+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:54.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:54.389+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:54.527 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:37:55.231 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:37:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:37:55.107+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 32463 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:37:55.231 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:54.902+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:55.232 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:54.902+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:55.232 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:54.902+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:55.232 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:54.903+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:55.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:55.411+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:55.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:55.411+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:55.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:55.411+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:55.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:55.411+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:55.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:55.232+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:55.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:55.232+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:55.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:55.232+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:55.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:55.233+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:56.244 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:55.934+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:56.244 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:55.934+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:56.244 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:55.934+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:56.244 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:55.935+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:56.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:56.376+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:56.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:56.376+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:56.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:56.376+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:56.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:56.377+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:56.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:56.245+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:56.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:56.245+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:56.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:56.245+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:56.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:56.246+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:57.268 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:56.920+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:57.268 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:56.920+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:57.269 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:56.920+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:57.269 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:56.921+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:57.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:57.337+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:57.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:57.337+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:57.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:57.337+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:57.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:57.338+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:57.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:57.269+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:57.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:57.269+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:57.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:57.269+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:57.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:57.270+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:58.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:57.922+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:58.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:57.922+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:58.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:57.922+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:58.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:57.923+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:58.703 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:58.333+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:58.703 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:58.333+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:58.704 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:58.333+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:58.704 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:58.334+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:58.704 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:58.317+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:58.704 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:58.317+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:58.704 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:58.317+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:58.704 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:58.318+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:59.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:58.964+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:59.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:58.964+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:59.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:58.964+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:37:59.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:58.965+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:59.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:59.318+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:59.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:59.318+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:59.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:59.318+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:37:59.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:37:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:37:59.318+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:37:59.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:59.308+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:59.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:59.308+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:59.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:59.308+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:37:59.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:37:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:37:59.309+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:00.232 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:38:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:38:00.112+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 32546 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:38:00.233 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:59.963+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:00.233 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:59.963+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:00.233 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:59.963+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:00.233 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:37:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:37:59.964+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:00.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:00.277+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:00.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:00.277+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:00.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:00.277+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:00.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:00.278+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:00.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:00.344+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:00.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:00.344+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:00.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:00.344+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:00.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:00.345+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:01.259 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:00.936+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:01.259 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:00.936+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:01.260 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:00.936+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:01.260 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:00.937+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:01.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:01.391+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:01.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:01.391+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:01.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:01.391+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:01.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:01.391+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:01.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:01.260+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:01.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:01.260+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:01.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:01.260+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:01.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:01.261+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:02.017 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:01.963+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:02.017 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:01.963+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:02.017 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:01.963+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:02.017 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:01.963+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:02.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:02.214+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:02.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:02.214+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:02.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:02.214+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:02.275 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:02.215+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:02.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:02.387+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:02.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:02.387+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:02.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:02.387+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:02.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:02.388+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:03.237 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:02.948+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:03.238 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:02.948+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:03.238 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:02.948+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:03.238 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:02.948+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:03.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:03.432+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:03.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:03.432+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:03.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:03.432+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:03.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:03.433+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:03.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:03.238+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:03.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:03.238+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:03.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:03.238+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:03.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:03.239+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:04.241 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:03.948+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:04.241 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:03.948+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:04.241 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:03.948+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:04.241 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:03.949+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:04.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:04.436+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:04.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:04.436+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:04.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:04.436+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:04.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:04.436+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:04.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:04.241+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:04.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:04.241+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:04.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:04.241+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:04.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:04.242+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:05.236 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:38:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:38:05.116+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 32631 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:38:05.236 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:04.908+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:05.236 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:04.908+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:05.237 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:04.908+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:05.237 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:04.909+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:05.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:05.476+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:05.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:05.476+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:05.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:05.476+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:05.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:05.477+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:05.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:05.236+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:05.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:05.236+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:05.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:05.236+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:05.526 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:05.237+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:06.271 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:05.910+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:06.272 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:05.910+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:06.272 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:05.910+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:06.272 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:05.911+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:06.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:06.272+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:06.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:06.272+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:06.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:06.272+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:06.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:06.273+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:06.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:06.502+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:06.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:06.502+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:06.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:06.502+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:06.526 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:06.502+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:06.527 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:38:06.528 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:38:07.054 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:06.959+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:07.054 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:06.959+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:07.054 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:06.959+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:07.054 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:06.960+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:07.382 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:07.296+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:07.382 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:07.296+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:07.382 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:07.296+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:07.383 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:07.297+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:07.651 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:07.509+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:07.652 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:07.509+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:07.652 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:07.509+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:07.652 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:07.510+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:08.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:07.940+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:08.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:07.940+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:08.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:07.940+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:08.276 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:07.941+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:08.686 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:08.540+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:08.687 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:08.540+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:08.687 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:08.540+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:08.687 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:08.541+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:08.687 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:08.346+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:08.687 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:08.346+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:08.688 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:08.346+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:08.688 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:08.347+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:09.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:08.930+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:09.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:08.930+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:09.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:08.930+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:09.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:08.931+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:09.598 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:38:09.654 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:09.375+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:09.654 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:09.375+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:09.654 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:09.375+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:09.655 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:09.375+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:09.655 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:09.514+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:09.655 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:09.514+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:09.655 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:09.514+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:09.655 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:09.515+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:10.241 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:38:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:38:10.120+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 32712 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:38:10.242 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:09.955+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:10.242 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:09.955+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:10.242 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:09.955+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:10.242 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:09.956+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:10.503 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:10.377+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:10.503 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:10.377+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:10.503 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:10.377+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:10.503 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:10.377+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:10.774 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:10.505+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:10.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:10.505+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:10.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:10.505+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:10.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:10.505+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:11.233 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:10.971+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:11.233 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:10.971+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:11.233 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:10.971+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:11.233 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:10.972+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:11.485 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:11.415+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:11.485 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:11.415+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:11.486 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:11.415+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:11.486 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:11.416+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:11.765 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:11.486+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:11.765 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:11.486+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:11.765 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:11.486+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:11.765 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:11.487+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:12.024 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:11.935+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:12.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:11.935+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:12.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:11.935+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:12.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:11.936+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:12.599 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:38:12.600 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:38:12.670 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:38:12.670 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-11T23:38:12.671 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-11T23:38:12.673 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592012/remote/smithi153/crash 2024-03-11T23:38:12.674 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/crash -- . 2024-03-11T23:38:12.701 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:12.516+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:12.702 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:12.516+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:12.702 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:12.516+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:12.702 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:12.517+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:12.702 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:12.435+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:12.702 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:12.435+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:12.702 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:12.435+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:12.703 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:12.436+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:12.706 DEBUG:teuthology.misc:Transferring archived files from smithi163:/var/lib/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592012/remote/smithi163/crash 2024-03-11T23:38:12.707 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:38:12.707 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:38:13.024 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:12.942+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:13.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:12.942+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:13.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:12.942+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:13.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:12.943+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:13.645 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:13.390+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:13.645 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:13.390+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:13.645 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:13.390+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:13.645 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:13.391+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:13.646 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:13.483+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:13.646 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:13.483+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:13.646 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:13.483+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:13.646 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:13.483+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:14.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:13.914+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:14.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:13.914+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:14.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:13.914+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:14.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:13.915+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:14.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:14.484+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:14.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:14.484+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:14.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:14.484+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:14.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:14.485+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:14.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:14.416+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:14.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:14.416+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:14.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:14.416+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:14.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:14.417+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:15.245 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:38:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:38:15.124+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 32798 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:38:15.246 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:14.956+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:15.246 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:14.956+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:15.246 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:14.956+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:15.246 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:14.957+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:15.523 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:15.441+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:15.523 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:15.441+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:15.524 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:15.441+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:15.524 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:15.442+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:15.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:15.524+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:15.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:15.524+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:15.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:15.524+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:15.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:15.525+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:15.778 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:38:16.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:15.917+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:16.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:15.917+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:16.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:15.917+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:16.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:15.917+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:16.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:16.476+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:16.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:16.476+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:16.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:16.476+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:16.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:16.477+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:16.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:16.448+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:16.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:16.448+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:16.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:16.448+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:16.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:16.449+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:17.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:16.877+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:17.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:16.877+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:17.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:16.877+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:17.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:16.878+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:17.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:17.438+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:17.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:17.438+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:17.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:17.438+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:17.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:17.438+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:17.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:17.501+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:17.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:17.501+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:17.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:17.501+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:17.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:17.501+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:18.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:17.877+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:18.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:17.877+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:18.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:17.877+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:18.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:17.878+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:18.757 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:18.492+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:18.757 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:18.492+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:18.757 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:18.492+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:18.757 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:18.493+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:18.758 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:18.412+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:18.758 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:18.412+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:18.758 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:18.412+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:18.758 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:18.413+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:19.024 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:18.896+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:19.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:18.896+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:19.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:18.896+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:19.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:18.897+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:19.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:19.495+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:19.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:19.495+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:19.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:19.495+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:19.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:19.495+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:19.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:19.398+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:19.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:19.398+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:19.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:19.398+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:19.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:19.399+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:20.128 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:19.877+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:20.128 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:19.877+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:20.128 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:19.877+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:20.129 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:19.878+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:20.435 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:38:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:38:20.129+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 32881 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:38:20.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:20.531+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:20.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:20.531+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:20.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:20.531+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:20.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:20.532+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:20.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:20.436+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:20.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:20.436+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:20.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:20.436+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:20.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:20.437+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:21.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:20.836+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:21.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:20.836+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:21.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:20.836+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:21.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:20.837+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:21.765 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:21.419+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:21.765 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:21.419+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:21.766 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:21.419+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:21.766 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:21.421+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:21.766 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:21.502+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:21.766 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:21.502+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:21.766 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:21.502+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:21.766 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:21.502+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:22.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:21.825+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:22.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:21.825+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:22.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:21.825+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:22.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:21.825+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:22.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:22.420+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:22.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:22.420+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:22.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:22.420+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:22.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:22.421+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:22.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:22.499+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:22.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:22.499+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:22.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:22.499+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:22.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:22.500+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:23.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:22.785+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:23.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:22.785+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:23.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:22.785+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:23.275 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:22.786+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:23.646 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:23.521+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:23.646 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:23.521+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:23.646 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:23.521+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:23.647 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:23.522+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:23.647 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:23.389+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:23.647 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:23.389+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:23.647 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:23.389+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:23.647 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:23.390+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:24.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:23.751+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:24.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:23.751+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:24.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:23.751+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:24.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:23.751+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:24.737 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:24.427+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:24.737 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:24.427+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:24.737 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:24.427+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:24.737 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:24.429+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:24.738 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:24.518+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:24.738 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:24.518+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:24.738 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:24.518+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:24.738 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:24.519+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:24.779 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:38:24.779 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:38:25.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:24.738+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:25.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:24.738+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:25.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:24.738+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:25.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:24.739+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:25.441 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:38:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:38:25.133+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 32966 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:38:25.734 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:25.443+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:25.734 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:25.443+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:25.734 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:25.443+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:25.734 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:25.444+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:25.735 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:25.485+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:25.735 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:25.485+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:25.735 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:25.485+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:25.735 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:25.486+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:26.024 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:25.735+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:26.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:25.735+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:26.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:25.735+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:26.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:25.736+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:26.698 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:26.429+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:26.698 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:26.429+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:26.699 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:26.429+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:26.699 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:26.429+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:26.700 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:26.444+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:26.700 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:26.444+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:26.700 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:26.444+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:26.700 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:26.446+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:27.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:26.698+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:27.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:26.698+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:27.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:26.698+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:27.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:26.700+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:27.654 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:27.494+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:27.654 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:27.494+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:27.654 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:27.494+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:27.654 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:27.495+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:27.654 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:27.387+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:27.655 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:27.387+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:27.655 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:27.387+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:27.655 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:27.388+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:27.838 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:38:28.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:27.654+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:28.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:27.654+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:28.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:27.654+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:28.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:27.655+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:28.674 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:28.387+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:28.674 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:28.387+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:28.674 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:28.387+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:28.674 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:28.388+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:28.675 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:28.531+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:28.675 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:28.531+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:28.675 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:28.531+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:28.675 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:28.532+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:29.024 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:28.675+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:29.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:28.675+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:29.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:28.675+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:29.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:28.676+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:29.638 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:29.529+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:29.639 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:29.529+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:29.639 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:29.529+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:29.639 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:29.530+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:29.639 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:29.381+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:29.639 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:29.381+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:29.639 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:29.381+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:29.639 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:29.381+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:30.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:29.639+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:30.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:29.639+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:30.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:29.639+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:30.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:29.640+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:30.389 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:38:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:38:30.137+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 33049 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:38:30.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:30.484+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:30.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:30.484+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:30.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:30.484+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:30.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:30.484+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:30.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:30.592+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:30.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:30.592+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:30.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:30.592+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:30.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:30.593+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:30.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:30.391+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:30.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:30.391+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:30.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:30.391+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:30.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:30.391+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:31.766 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:31.393+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:31.766 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:31.393+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:31.766 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:31.393+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:31.766 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:31.394+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:31.767 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:31.576+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:31.767 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:31.576+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:31.767 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:31.576+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:31.767 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:31.576+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:31.768 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:31.470+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:31.768 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:31.470+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:31.768 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:31.470+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:31.768 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:31.470+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:32.737 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:32.547+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:32.737 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:32.547+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:32.737 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:32.547+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:32.737 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:32.548+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:32.738 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:32.422+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:32.738 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:32.422+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:32.738 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:32.422+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:32.738 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:32.423+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:32.738 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:32.428+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:32.739 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:32.428+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:32.739 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:32.428+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:32.739 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:32.429+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:33.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:33.450+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:33.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:33.450+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:33.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:33.450+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:33.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:33.451+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:33.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:33.578+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:33.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:33.578+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:33.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:33.578+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:33.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:33.578+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:33.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:33.461+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:33.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:33.461+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:33.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:33.461+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:33.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:33.462+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:34.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:34.433+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:34.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:34.433+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:34.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:34.433+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:34.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:34.434+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:34.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:34.509+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:34.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:34.509+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:34.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:34.509+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:34.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:34.510+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:34.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:34.543+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:34.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:34.543+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:34.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:34.543+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:34.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:34.543+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:35.394 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:38:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:38:35.141+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 33134 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:38:35.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:35.395+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:35.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:35.395+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:35.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:35.395+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:35.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:35.396+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:35.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:35.534+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:35.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:35.534+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:35.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:35.534+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:35.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:35.535+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:35.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:35.565+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:35.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:35.565+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:35.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:35.565+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:35.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:35.566+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:36.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:36.503+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:36.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:36.503+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:36.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:36.503+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:36.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:36.503+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:36.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:36.591+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:36.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:36.591+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:36.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:36.591+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:36.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:36.592+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:36.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:36.399+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:36.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:36.399+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:36.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:36.399+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:36.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:36.400+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:37.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:37.400+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:37.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:37.400+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:37.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:37.400+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:37.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:37.400+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:37.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:37.471+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:37.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:37.471+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:37.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:37.471+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:37.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:37.472+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:37.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:37.628+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:37.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:37.628+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:37.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:37.628+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:37.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:37.629+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:38.774 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:38.507+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:38.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:38.507+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:38.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:38.507+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:38.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:38.508+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:38.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:38.580+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:38.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:38.580+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:38.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:38.580+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:38.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:38.581+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:38.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:38.435+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:38.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:38.435+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:38.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:38.435+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:38.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:38.437+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:39.676 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:39.468+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:39.676 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:39.468+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:39.677 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:39.468+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:39.677 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:39.469+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:39.677 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:39.468+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:39.677 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:39.468+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:39.677 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:39.468+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:39.677 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:39.469+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:39.678 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:39.573+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:39.678 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:39.573+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:39.678 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:39.573+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:39.678 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:39.573+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:39.839 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:38:39.840 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:38:40.458 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:38:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:38:40.146+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 33217 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:38:40.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:40.583+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:40.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:40.583+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:40.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:40.583+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:40.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:40.583+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:40.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:40.509+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:40.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:40.509+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:40.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:40.509+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:40.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:40.510+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:40.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:40.459+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:40.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:40.459+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:40.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:40.459+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:40.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:40.460+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:41.768 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:41.609+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:41.768 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:41.609+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:41.768 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:41.609+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:41.768 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:41.610+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:41.769 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:41.461+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:41.769 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:41.461+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:41.769 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:41.461+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:41.769 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:41.462+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:41.769 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:41.472+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:41.770 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:41.472+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:41.770 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:41.472+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:41.770 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:41.473+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:42.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:42.462+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:42.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:42.462+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:42.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:42.462+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:42.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:42.463+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:42.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:42.513+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:42.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:42.513+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:42.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:42.513+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:42.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:42.514+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:42.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:42.593+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:42.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:42.593+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:42.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:42.593+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:42.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:42.594+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:42.910 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:38:43.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:43.552+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:43.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:43.552+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:43.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:43.552+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:43.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:43.553+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:43.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:43.467+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:43.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:43.467+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:43.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:43.467+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:43.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:43.467+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:43.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:43.602+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:43.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:43.602+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:43.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:43.602+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:43.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:43.603+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:44.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:44.558+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:44.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:44.558+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:44.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:44.558+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:44.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:44.559+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:44.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:44.561+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:44.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:44.561+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:44.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:44.561+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:44.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:44.562+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:44.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:44.496+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:44.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:44.496+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:44.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:44.496+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:44.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:44.497+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:45.518 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:38:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:38:45.150+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 33302 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:38:45.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:45.519+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:45.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:45.519+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:45.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:45.519+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:45.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:45.520+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:45.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:45.542+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:45.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:45.542+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:45.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:45.542+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:45.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:45.543+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:45.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:45.599+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:45.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:45.599+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:45.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:45.599+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:45.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:45.600+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:45.911 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:38:45.912 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:38:45.982 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:38:45.982 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-11T23:38:45.983 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-11T23:38:45.983 DEBUG:teuthology.orchestra.run.smithi153:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-11T23:38:46.016 INFO:teuthology.orchestra.run.smithi153.stdout:1710198000.000206 mon.smithi153 (mon.0) 646 : cluster [ERR] Health detail: HEALTH_ERR 1 filesystem is offline; 1 filesystem is online with fewer MDS than max_mds 2024-03-11T23:38:46.016 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-11T23:38:46.016 INFO:tasks.cephadm:Compressing logs... 2024-03-11T23:38:46.017 DEBUG:teuthology.orchestra.run.smithi153:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T23:38:46.059 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:38:46.059 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:38:46.085 INFO:teuthology.orchestra.run.smithi153.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T23:38:46.085 INFO:teuthology.orchestra.run.smithi153.stderr:: No such file or directory 2024-03-11T23:38:46.085 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph-mon.smithi153.log 2024-03-11T23:38:46.085 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T23:38:46.085 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph-osd.1.log 2024-03-11T23:38:46.085 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph-mon.smithi153.log: /var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph-mds.cephfs.smithi153.yuzdaj.log 2024-03-11T23:38:46.085 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph-mgr.smithi153.nyocep.log 2024-03-11T23:38:46.086 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph-mds.cephfs.smithi153.yuzdaj.log: gzip -5 --verbose -- /var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph-mds.cephfs.smithi153.qxjyxz.log 2024-03-11T23:38:46.086 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph-mgr.smithi153.nyocep.log: 90.0% -- replaced with /var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T23:38:46.087 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph-osd.0.log 2024-03-11T23:38:46.087 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph-mds.cephfs.smithi153.qxjyxz.log: gzip -5 --verbose -- /var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T23:38:46.089 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph.log 2024-03-11T23:38:46.090 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph.cephadm.log: 81.5% -- replaced with /var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T23:38:46.092 INFO:teuthology.orchestra.run.smithi153.stderr: 93.3%gzip -5 --verbose -- /var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph-osd.2.log 2024-03-11T23:38:46.092 INFO:teuthology.orchestra.run.smithi153.stderr: -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T23:38:46.093 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T23:38:46.094 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph-osd.2.log: 85.9% -- replaced with /var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T23:38:46.117 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph-volume.log: 90.8% -- replaced with /var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T23:38:46.956 INFO:teuthology.orchestra.run.smithi153.stderr:gzip: /var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph-mgr.smithi153.nyocep.log: file size changed while zipping 2024-03-11T23:38:46.956 INFO:teuthology.orchestra.run.smithi153.stderr: 90.0% -- replaced with /var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph-mgr.smithi153.nyocep.log.gz 2024-03-11T23:38:47.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:46.564+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:47.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:46.564+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:47.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:46.564+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:47.026 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:46.565+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:47.026 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:46.525+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:47.026 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:46.525+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:47.026 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:46.525+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:47.027 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:46.526+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:47.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:46.615+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:47.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:46.615+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:47.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:46.615+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:47.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:46.616+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:47.513 INFO:teuthology.orchestra.run.smithi153.stderr:gzip: /var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph-mon.smithi153.log: file size changed while zipping 2024-03-11T23:38:47.513 INFO:teuthology.orchestra.run.smithi153.stderr: 92.7% -- replaced with /var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph-mon.smithi153.log.gz 2024-03-11T23:38:47.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:47.532+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:47.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:47.532+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:47.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:47.532+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:47.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:47.534+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:47.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:47.497+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:47.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:47.497+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:47.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:47.497+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:47.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:47.498+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:47.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:47.575+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:47.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:47.575+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:47.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:47.575+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:47.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:47.576+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:48.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:48.506+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:48.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:48.506+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:48.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:48.506+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:48.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:48.507+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:48.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:48.487+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:48.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:48.487+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:48.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:48.487+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:48.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:48.488+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:48.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:48.619+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:48.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:48.619+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:48.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:48.619+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:48.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:48.620+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:49.122 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:38:49.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:49.518+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:49.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:49.518+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:49.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:49.518+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:49.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:49.520+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:49.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:49.458+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:49.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:49.458+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:49.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:49.458+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:49.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:49.458+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:49.778 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:49.580+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:49.778 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:49.580+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:49.779 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:49.580+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:49.779 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:49.581+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:50.506 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:50.506+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:50.507 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:50.506+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:50.507 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:50.506+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:50.507 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:50.507+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:50.507 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:38:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:38:50.154+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 33385 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:38:50.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:50.576+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:50.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:50.576+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:50.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:50.576+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:50.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:50.577+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:50.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:50.527+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:50.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:50.527+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:50.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:50.527+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:50.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:50.527+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:51.485 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:51.486+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:51.485 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:51.486+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:51.485 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:51.486+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:51.768 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:51.569+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:51.768 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:51.569+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:51.768 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:51.569+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:51.768 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:51.570+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:51.769 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:51.487+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:51.769 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:51.564+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:51.769 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:51.564+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:51.769 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:51.564+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:51.769 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:51.565+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:52.221 INFO:teuthology.orchestra.run.smithi153.stderr:gzip: /var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph-osd.0.log: file size changed while zipping 2024-03-11T23:38:52.221 INFO:teuthology.orchestra.run.smithi153.stderr: 95.0% -- replaced with /var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph-osd.0.log.gz 2024-03-11T23:38:52.322 INFO:teuthology.orchestra.run.smithi153.stderr:gzip: /var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph-osd.2.log: file size changed while zipping 2024-03-11T23:38:52.322 INFO:teuthology.orchestra.run.smithi153.stderr: 95.0% -- replaced with /var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph-osd.2.log.gz 2024-03-11T23:38:52.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:52.522+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:52.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:52.522+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:52.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:52.522+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:52.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:52.523+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:52.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:52.588+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:52.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:52.588+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:52.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:52.588+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:52.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:52.589+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:52.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:52.558+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:52.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:52.558+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:52.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:52.558+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:52.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:52 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:52.559+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:53.484 INFO:teuthology.orchestra.run.smithi153.stderr: 93.3% -- replaced with /var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph-mds.cephfs.smithi153.yuzdaj.log.gz 2024-03-11T23:38:53.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:53.599+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:53.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:53.599+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:53.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:53.599+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:53.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:53.600+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:53.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:53.579+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:53.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:53.579+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:53.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:53.579+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:53.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:53.581+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:53.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:53.523+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:53.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:53.523+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:53.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:53.523+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:53.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:53 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:53.524+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:55.025 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:54.565+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:55.025 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:54.565+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:55.025 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:54.565+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:55.025 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:54.566+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:55.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:54.621+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:55.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:54.621+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:55.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:54.621+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:55.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:54.622+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:55.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:54.541+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:55.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:54.541+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:55.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:54.541+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:55.027 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:54 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:54.542+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:55.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:38:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:38:55.158+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 33470 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:38:55.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:55.609+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:55.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:55.609+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:55.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:55.609+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:55.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:55.610+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:55.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:55.503+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:55.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:55.503+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:55.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:55.503+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:55.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:55.504+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:55.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:55.550+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:55.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:55.550+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:55.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:55.550+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:55.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:55 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:55.551+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:56.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:56.475+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:56.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:56.475+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:56.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:56.475+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:56.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:56.476+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:56.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:56.553+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:56.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:56.553+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:56.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:56.553+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:56.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:56.554+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:56.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:56.619+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:56.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:56.619+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:56.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:56.619+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:56.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:56 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:56.620+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:57.681 INFO:teuthology.orchestra.run.smithi153.stderr:gzip: /var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph-osd.1.log: file size changed while zipping 2024-03-11T23:38:57.681 INFO:teuthology.orchestra.run.smithi153.stderr: 97.0% -- replaced with /var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph-osd.1.log.gz 2024-03-11T23:38:57.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:57.437+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:57.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:57.437+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:57.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:57.437+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:57.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:57.438+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:57.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:57.511+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:57.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:57.511+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:57.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:57.511+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:57.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:57.512+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:57.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:57.610+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:57.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:57.610+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:57.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:57.610+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:57.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:57 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:57.611+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:58.123 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:38:58.124 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:38:58.774 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:58.644+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:58.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:58.644+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:58.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:58.644+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:58.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:58.645+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:58.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:58.457+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:58.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:58.457+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:58.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:58.457+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:58.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:58.458+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:58.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:58.491+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:58.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:58.491+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:58.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:58.491+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:58.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:58 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:58.492+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:59.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:59.601+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:59.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:59.601+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:59.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:59.601+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:38:59.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:38:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:38:59.602+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:59.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:59.464+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:59.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:59.464+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:59.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:59.464+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:38:59.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:38:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:38:59.465+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:38:59.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:59.474+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:59.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:59.474+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:59.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:59.474+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:38:59.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:38:59 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:38:59.474+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:00.445 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:39:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:39:00.162+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 33553 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:39:00.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:00.446+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:00.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:00.446+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:00.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:00.446+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:00.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:00.447+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:00.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:00.503+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:00.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:00.503+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:00.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:00.503+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:00.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:00.503+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:00.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:00.553+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:00.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:00.554+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:00.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:00.554+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:00.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:00 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:00.555+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:01.182 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:39:01.769 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:01.543+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:01.769 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:01.543+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:01.769 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:01.543+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:01.769 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:01.544+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:01.770 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:01.399+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:01.770 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:01.399+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:01.770 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:01.399+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:01.770 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:01.400+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:01.770 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:01.517+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:01.771 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:01.517+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:01.771 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:01.517+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:01.771 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:01 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:01.518+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:02.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:02.365+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:02.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:02.365+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:02.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:02.365+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:02.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:02.365+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:02.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:02.512+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:02.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:02.512+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:02.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:02.512+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:02.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:02.513+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:02.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:02.584+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:02.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:02.584+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:02.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:02.584+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:02.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:02 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:02.585+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:03.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:03.406+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:03.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:03.406+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:03.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:03.406+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:03.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:03.407+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:03.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:03.472+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:03.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:03.472+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:03.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:03.472+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:03.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:03.473+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:03.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:03.611+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:03.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:03.611+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:03.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:03.611+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:03.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:03 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:03.612+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:03.887 INFO:teuthology.orchestra.run.smithi153.stderr: 92.2% -- replaced with /var/log/ceph/fa913b96-dff9-11ee-95c8-87774f69a715/ceph-mds.cephfs.smithi153.qxjyxz.log.gz 2024-03-11T23:39:03.889 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-11T23:39:03.889 INFO:teuthology.orchestra.run.smithi153.stderr:real 0m17.820s 2024-03-11T23:39:03.890 INFO:teuthology.orchestra.run.smithi153.stderr:user 0m49.965s 2024-03-11T23:39:03.890 INFO:teuthology.orchestra.run.smithi153.stderr:sys 0m1.253s 2024-03-11T23:39:04.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:04.426+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:04.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:04.426+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:04.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:04.426+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:04.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:04.427+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:04.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:04.510+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:04.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:04.510+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:04.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:04.510+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:04.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:04.511+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:04.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:04.589+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:04.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:04.589+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:04.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:04.589+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:04.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:04 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:04.589+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:05.454 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:39:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:39:05.166+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 33638 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:39:05.774 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:05.455+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:05.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:05.455+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:05.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:05.455+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:05.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:05.456+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:05.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:05.511+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:05.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:05.511+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:05.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:05.511+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:05.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:05.512+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:05.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:05.605+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:05.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:05.605+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:05.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:05.605+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:05.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:05 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:05.606+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:06.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:06.446+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:06.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:06.446+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:06.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:06.446+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:06.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:06.447+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:06.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:06.576+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:06.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:06.576+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:06.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:06.576+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:06.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:06.577+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:06.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:06.469+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:06.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:06.469+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:06.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:06.469+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:06.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:06 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:06.469+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:07.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:07.481+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:07.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:07.481+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:07.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:07.481+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:07.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:07.483+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:07.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:07.583+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:07.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:07.583+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:07.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:07.583+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:07.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:07.585+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:07.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:07.436+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:07.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:07.436+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:07.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:07.436+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:07.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:07 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:07.436+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:08.774 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:08.474+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:08.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:08.474+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:08.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:08.474+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:08.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:08.474+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:08.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:08.464+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:08.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:08.464+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:08.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:08.464+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:08.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:08.465+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:08.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:08.552+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:08.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:08.552+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:08.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:08.552+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:08.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:08 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:08.552+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:09.676 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:09.507+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:09.676 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:09.507+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:09.676 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:09.507+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:09.676 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:09.508+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:09.676 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:09.477+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:09.677 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:09.477+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:09.677 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:09.477+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:09.677 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:09.478+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:09.677 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:09.493+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:09.677 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:09.493+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:09.677 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:09.493+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:09.678 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:09 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:09.494+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:10.467 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:39:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:39:10.171+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 33721 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:39:10.774 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:10.469+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:10.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:10.469+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:10.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:10.469+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:10.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:10.470+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:10.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:10.475+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:10.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:10.476+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:10.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:10.476+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:10.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:10.476+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:10.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:10.508+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:10.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:10.508+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:10.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:10.508+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:10.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:10 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:10.509+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:11.769 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:11.499+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:11.770 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:11.499+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:11.770 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:11.499+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:11.770 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:11.499+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:11.770 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:11.496+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:11.771 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:11.496+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:11.771 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:11.496+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:11.771 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:11.497+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:11.771 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:11.480+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:11.771 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:11.480+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:11.771 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:11.480+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:11.772 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:11 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:11.481+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:12.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:12.458+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:12.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:12.458+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:12.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:12.458+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:12.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:12.459+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:12.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:12.516+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:12.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:12.516+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:12.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:12.516+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:12.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:12.517+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:12.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:12.536+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:12.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:12.536+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:12.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:12.536+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:12.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:12 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:12.537+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:13.183 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:39:13.183 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:39:13.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:13.523+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:13.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:13.523+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:13.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:13.523+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:13.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:13.523+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:13.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:13.463+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:13.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:13.463+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:13.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:13.463+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:13.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:13.464+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:13.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:13.488+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:13.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:13.488+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:13.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:13.488+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:13.778 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:13 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:13.489+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:14.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:14.460+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:14.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:14.460+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:14.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:14.460+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:14.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:14.461+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:14.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:14.557+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:14.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:14.557+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:14.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:14.557+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:14.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:14.558+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:14.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:14.505+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:14.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:14.505+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:14.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:14.505+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:14.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:14 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:14.506+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:15.453 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:39:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:39:15.175+0000 7f0da1c25700 -1 mon.smithi153@0(probing) e2 get_health_metrics reporting 33806 slow ops, oldest is monmgrreport(gid 14204, 0 checks, 1 progress events) 2024-03-11T23:39:15.745 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:15.532+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:15.745 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:15.532+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:15.745 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:15.532+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:15.746 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:15.533+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:15.746 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:15.499+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:15.746 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:15.499+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:15.746 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:15.499+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:15.746 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:15.499+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:15.747 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:15.455+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:15.747 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:15.455+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:15.747 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:15.455+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:15.747 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:15 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:15.455+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:16.258 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:39:16.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:16.533+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:16.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:16.533+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:16.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:16.533+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:16.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:16.533+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:16.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:16.484+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:16.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:16.484+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:16.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:16.484+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:16.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:16.485+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:16.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:16.557+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:16.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:16.557+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:16.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:16.557+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:16.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:16 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:16.558+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:17.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:17.511+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:17.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:17.511+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:17.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:17.511+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:17.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:17.512+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:17.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:17.569+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:17.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:17.569+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:17.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:17.569+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:17.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:17.570+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:17.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:17.493+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:17.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:17.493+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:17.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:17.493+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:17.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:17 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:17.494+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:19.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:18.531+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:19.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:18.531+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:19.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:18.531+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:19.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:18.531+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:19.026 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:18.542+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:19.026 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:18.542+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:19.026 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:18.542+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:19.026 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:18.543+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:19.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:18.587+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:19.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:18.587+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:19.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:18.587+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:19.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:18 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:18.588+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:19.259 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:39:19.260 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:39:19.326 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:39:19.326 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-11T23:39:19.327 INFO:tasks.cephadm:Removing cluster... 2024-03-11T23:39:19.327 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid fa913b96-dff9-11ee-95c8-87774f69a715 --force 2024-03-11T23:39:19.626 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:19.598+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:19.626 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:19.598+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:19.626 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:19.598+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:19.626 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:19.599+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:19.626 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:19.498+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:19.627 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:19.498+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:19.627 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:19.498+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:19.627 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:19.499+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:19.627 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:19.579+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:19.627 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:19.579+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:19.627 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:19.579+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:19.628 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:19.579+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:19.914 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:39:19 smithi153 systemd[1]: Stopping Ceph mon.smithi153 for fa913b96-dff9-11ee-95c8-87774f69a715... 2024-03-11T23:39:20.199 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:39:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:39:19.915+0000 7f0da4c2b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi153 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T23:39:20.200 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:39:19 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153[28958]: 2024-03-11T23:39:19.915+0000 7f0da4c2b700 -1 mon.smithi153@0(probing) e2 *** Got Signal Terminated *** 2024-03-11T23:39:20.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:39:20 smithi153 podman[84451]: 2024-03-11 23:39:20.322085667 +0000 UTC m=+0.534434190 container died 0dad3ee1fdfffa6fa2e3579fca46d272ad782cee39b501b8a50a1c6a88af8bdf (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-c186d0c, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T23:39:20.766 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:20.575+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:20.766 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:20.575+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:20.766 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:20.575+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:20.766 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:20.575+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:20.767 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:20.604+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:20.767 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:20.604+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:20.767 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:20.604+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:20.767 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:20.605+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:20.767 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:20.503+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:20.767 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:20.503+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:20.767 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:20.504+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:20.768 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:20 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:20.505+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:21.025 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:39:20 smithi153 podman[84451]: 2024-03-11 23:39:20.867208943 +0000 UTC m=+1.079557467 container cleanup 0dad3ee1fdfffa6fa2e3579fca46d272ad782cee39b501b8a50a1c6a88af8bdf (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, RELEASE=quincy-c186d0c, CEPH_POINT_RELEASE=) 2024-03-11T23:39:21.025 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:39:20 smithi153 bash[84451]: ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153 2024-03-11T23:39:21.334 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:39:21 smithi153 podman[84461]: 2024-03-11 23:39:21.077116055 +0000 UTC m=+0.754953106 container remove 0dad3ee1fdfffa6fa2e3579fca46d272ad782cee39b501b8a50a1c6a88af8bdf (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-fa913b96-dff9-11ee-95c8-87774f69a715-mon-smithi153, RELEASE=quincy-c186d0c, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-11T23:39:21.601 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:21.588+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:21.602 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:21.588+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:21.602 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:21.589+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:21.602 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:21.589+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:21.602 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:39:21 smithi153 systemd[1]: ceph-fa913b96-dff9-11ee-95c8-87774f69a715@mon.smithi153.service: Deactivated successfully. 2024-03-11T23:39:21.602 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:39:21 smithi153 systemd[1]: Stopped Ceph mon.smithi153 for fa913b96-dff9-11ee-95c8-87774f69a715. 2024-03-11T23:39:21.602 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Mar 11 23:39:21 smithi153 systemd[1]: ceph-fa913b96-dff9-11ee-95c8-87774f69a715@mon.smithi153.service: Consumed 23.694s CPU time. 2024-03-11T23:39:21.603 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:21.474+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:21.603 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:21.474+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:21.603 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:21.474+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:21.603 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:21.475+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:21.879 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:21.602+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:21.879 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:21.602+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:21.879 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:21.602+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:21.879 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:21 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:21.603+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:22.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:22.610+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:22.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:22.610+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:22.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:22.610+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:22.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:22.611+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:22.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:22.507+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:22.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:22.507+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:22.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:22.507+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:22.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:22.508+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:22.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:22.650+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:22.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:22.650+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:22.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:22.650+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:22.777 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:22 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:22.651+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:23.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:23.509+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:23.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:23.509+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:23.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:23.509+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:23.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:23.510+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:23.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:23.608+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:23.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:23.608+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:23.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:23.608+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:23.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:23.609+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:23.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:23.576+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:23.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:23.576+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:23.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:23.576+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:23.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:23 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:23.577+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:24.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:24.475+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:24.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:24.475+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:24.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:24.475+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:24.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:24.476+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:24.826 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:24.574+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:24.826 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:24.574+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:24.826 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:24.574+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:24.826 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:24.575+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:24.827 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:24.640+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:24.827 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:24.640+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:24.827 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:24.640+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:24.827 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:24 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:24.641+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:25.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:25.447+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:25.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:25.447+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:25.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:25.447+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:25.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:25.448+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:25.822 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:25.603+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:25.822 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:25.603+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:25.822 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:25.603+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:25.823 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:25.603+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:25.823 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:25.608+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:25.823 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:25.608+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:25.823 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:25.608+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:25.823 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:25 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:25.609+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:26.709 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:26.447+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:26.710 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:26.447+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:26.710 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:26.447+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:26.710 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:26.447+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:26.710 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:26.645+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:26.710 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:26.645+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:26.711 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:26.645+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:26.711 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:26.646+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:26.711 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:26.560+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:26.711 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:26.560+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:26.711 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:26.560+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:26.711 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:26 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:26.560+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:27.506 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:27.402+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:27.507 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:27.402+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:27.507 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:27.402+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:27.507 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:27.403+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:27.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:27.612+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:27.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:27.612+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:27.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:27.612+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:27.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:27.614+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:27.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:27.567+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:27.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:27.567+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:27.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:27.567+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:27.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:27 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:27.568+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:28.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:28.385+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:28.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:28.385+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:28.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:28.385+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:28.525 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:28.385+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:28.912 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:28.529+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:28.912 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:28.529+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:28.912 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:28.529+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:28.912 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:28.530+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:28.912 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:28.614+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:28.913 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:28.614+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:28.913 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:28.614+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:28.913 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:28 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:28.615+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:29.688 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:29.582+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:29.688 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:29.582+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:29.688 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:29.582+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:29.688 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:29.582+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:29.688 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:29.557+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:29.689 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:29.557+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:29.689 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:29.557+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:29.689 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:29.558+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:29.689 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:29.423+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:29.689 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:29.423+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:29.689 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:29.423+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:29.690 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:29.424+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:29.690 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:29 smithi153 systemd[1]: Stopping Ceph osd.0 for fa913b96-dff9-11ee-95c8-87774f69a715... 2024-03-11T23:39:30.274 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:29.976+0000 7f3924958700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T23:39:30.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:29.976+0000 7f3924958700 -1 osd.0 47 *** Got signal Terminated *** 2024-03-11T23:39:30.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:29 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:29.976+0000 7f3924958700 -1 osd.0 47 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T23:39:30.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:30.594+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:30.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:30.594+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:30.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:30.594+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:30.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:30.595+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:30.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:30.519+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:30.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:30.519+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:30.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:30.519+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:30.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:30.520+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:30.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:30.391+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:30.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:30.391+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:30.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:30.391+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:30.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:30 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:30.392+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:31.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:31.381+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:31.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:31.381+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:31.778 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:31.381+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:31.779 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:31.382+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:31.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:31.556+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:31.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:31.556+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:31.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:31.556+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:31.779 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:31.557+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:31.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:31.475+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:31.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:31.475+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:31.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:31.475+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:31.780 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:31 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:31.476+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:32.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:32.425+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:32.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:32.425+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:32.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:32.425+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:32.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:32.426+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:32.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:32.531+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:32.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:32.531+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:32.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:32.531+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:32.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:32.531+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:32.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:32.464+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:32.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:32.464+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:32.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:32.464+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:32.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:32 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:32.465+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:33.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:33.460+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:33.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:33.460+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:33.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:33.460+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:33.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:33.461+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:33.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:33.560+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:33.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:33.560+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:33.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:33.560+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:33.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:33.560+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:33.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:33.514+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:33.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:33.514+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:33.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:33.514+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:33.777 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:33 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:33.515+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:34.774 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:34.534+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:34.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:34.534+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:34.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:34.534+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:34.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:34.535+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:34.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:34.555+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:34.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:34.555+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:34.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:34.555+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:34.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:34.556+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:34.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:34.466+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:34.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:34.466+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:34.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:34.466+0000 7f391ef52700 -1 osd.0 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:06.499232+0000 (oldest deadline 2024-03-11T23:34:26.499232+0000) 2024-03-11T23:39:34.777 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:34 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0[47072]: 2024-03-11T23:39:34.467+0000 7f391ef52700 -1 osd.0 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3782 3.1f 3:fac4be2e:::100000038cd.00000000:head [write 0~17545 in=17545b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:35.718 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:35 smithi153 podman[85286]: 2024-03-11 23:39:35.405795565 +0000 UTC m=+5.551004692 container died e1d999ae7d94b43a5c1577067d3c2f22e070b1f2436dc3d89aecaac17ed206c2 (image=quay.ceph.io/ceph-ci/ceph@sha256:ebd35aa0069fa2bbcdd3e20c6fffec19bb7fa44bfa11066e4a72109a40218c32, name=ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=quincy-c186d0c, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-11T23:39:35.718 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:35.532+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:35.718 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:35.532+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:35.719 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:35.532+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:35.719 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:35.533+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:35.719 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:35.533+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:35.719 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:35.533+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:35.719 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:35.533+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:35.719 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:35 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:35.534+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:36.024 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:35 smithi153 podman[85286]: 2024-03-11 23:39:35.827868056 +0000 UTC m=+5.973077183 container cleanup e1d999ae7d94b43a5c1577067d3c2f22e070b1f2436dc3d89aecaac17ed206c2 (image=quay.ceph.io/ceph-ci/ceph@sha256:ebd35aa0069fa2bbcdd3e20c6fffec19bb7fa44bfa11066e4a72109a40218c32, name=ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=quincy-c186d0c) 2024-03-11T23:39:36.025 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:35 smithi153 bash[85286]: ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0 2024-03-11T23:39:36.513 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:36 smithi153 podman[85304]: 2024-03-11 23:39:36.062673719 +0000 UTC m=+0.656777116 container remove e1d999ae7d94b43a5c1577067d3c2f22e070b1f2436dc3d89aecaac17ed206c2 (image=quay.ceph.io/ceph-ci/ceph@sha256:ebd35aa0069fa2bbcdd3e20c6fffec19bb7fa44bfa11066e4a72109a40218c32, name=ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=quincy-c186d0c, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-11T23:39:36.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:36.575+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:36.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:36.575+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:36.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:36.575+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:36.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:36.576+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:36.775 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:36 smithi153 podman[85395]: 2024-03-11 23:39:36.418218903 +0000 UTC m=+0.017330565 image pull a92afbb1c2834deb6f1bddda3f3c554061dd3ec88e0495c17c1373d9018e466b quay.ceph.io/ceph-ci/ceph@sha256:ebd35aa0069fa2bbcdd3e20c6fffec19bb7fa44bfa11066e4a72109a40218c32 2024-03-11T23:39:36.776 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:36 smithi153 podman[85395]: 2024-03-11 23:39:36.555269994 +0000 UTC m=+0.154381646 container create 5f6412bc15b8ed37ac096e311a626e3e47efe1ef0a99374855cc9108329e7401 (image=quay.ceph.io/ceph-ci/ceph@sha256:ebd35aa0069fa2bbcdd3e20c6fffec19bb7fa44bfa11066e4a72109a40218c32, name=ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0-deactivate, RELEASE=quincy-c186d0c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T23:39:36.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:36.566+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:36.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:36.566+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:36.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:36.566+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:36.776 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:36 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:36.567+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:37.127 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:36 smithi153 podman[85395]: 2024-03-11 23:39:36.805583815 +0000 UTC m=+0.404695468 container init 5f6412bc15b8ed37ac096e311a626e3e47efe1ef0a99374855cc9108329e7401 (image=quay.ceph.io/ceph-ci/ceph@sha256:ebd35aa0069fa2bbcdd3e20c6fffec19bb7fa44bfa11066e4a72109a40218c32, name=ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=quincy-c186d0c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307) 2024-03-11T23:39:37.127 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:36 smithi153 podman[85395]: 2024-03-11 23:39:36.80971427 +0000 UTC m=+0.408825920 container start 5f6412bc15b8ed37ac096e311a626e3e47efe1ef0a99374855cc9108329e7401 (image=quay.ceph.io/ceph-ci/ceph@sha256:ebd35aa0069fa2bbcdd3e20c6fffec19bb7fa44bfa11066e4a72109a40218c32, name=ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , RELEASE=quincy-c186d0c, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-11T23:39:37.128 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:36 smithi153 podman[85395]: 2024-03-11 23:39:36.916911051 +0000 UTC m=+0.516022708 container attach 5f6412bc15b8ed37ac096e311a626e3e47efe1ef0a99374855cc9108329e7401 (image=quay.ceph.io/ceph-ci/ceph@sha256:ebd35aa0069fa2bbcdd3e20c6fffec19bb7fa44bfa11066e4a72109a40218c32, name=ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=quincy-c186d0c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-11T23:39:37.452 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:37 smithi153 podman[85395]: 2024-03-11 23:39:37.130015712 +0000 UTC m=+0.729127361 container died 5f6412bc15b8ed37ac096e311a626e3e47efe1ef0a99374855cc9108329e7401 (image=quay.ceph.io/ceph-ci/ceph@sha256:ebd35aa0069fa2bbcdd3e20c6fffec19bb7fa44bfa11066e4a72109a40218c32, name=ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-c186d0c, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-11T23:39:37.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:37.590+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:37.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:37.590+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:37.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:37.590+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:37.775 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:37.592+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:37.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:37.599+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:37.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:37.599+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:37.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:37.599+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:37.776 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:37 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:37.600+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:38.274 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:38 smithi153 systemd[1]: Stopping Ceph osd.1 for fa913b96-dff9-11ee-95c8-87774f69a715... 2024-03-11T23:39:38.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:37 smithi153 podman[85420]: 2024-03-11 23:39:37.804690373 +0000 UTC m=+0.668671862 container remove 5f6412bc15b8ed37ac096e311a626e3e47efe1ef0a99374855cc9108329e7401 (image=quay.ceph.io/ceph-ci/ceph@sha256:ebd35aa0069fa2bbcdd3e20c6fffec19bb7fa44bfa11066e4a72109a40218c32, name=ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=quincy-c186d0c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-11T23:39:38.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:37 smithi153 systemd[1]: ceph-fa913b96-dff9-11ee-95c8-87774f69a715@osd.0.service: Deactivated successfully. 2024-03-11T23:39:38.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:37 smithi153 systemd[1]: Stopped Ceph osd.0 for fa913b96-dff9-11ee-95c8-87774f69a715. 2024-03-11T23:39:38.275 INFO:journalctl@ceph.osd.0.smithi153.stdout:Mar 11 23:39:37 smithi153 systemd[1]: ceph-fa913b96-dff9-11ee-95c8-87774f69a715@osd.0.service: Consumed 18min 9.362s CPU time. 2024-03-11T23:39:38.610 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:38.332+0000 7f9e68cab700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T23:39:38.611 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:38.332+0000 7f9e68cab700 -1 osd.1 47 *** Got signal Terminated *** 2024-03-11T23:39:38.611 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:38.332+0000 7f9e68cab700 -1 osd.1 47 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T23:39:39.024 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:38.627+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:39.025 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:38.627+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:39.025 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:38.627+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:39.025 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:38.628+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:39.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:38.611+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:39.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:38.611+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:39.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:38.611+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:39.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:38 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:38.612+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:40.024 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:39.670+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:40.025 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:39.670+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:40.025 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:39.670+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:40.025 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:39.670+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:40.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:39.600+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:40.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:39.600+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:40.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:39.600+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:40.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:39 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:39.601+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:41.024 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:40.674+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:41.025 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:40.674+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:41.025 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:40.674+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:41.025 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:40.675+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:41.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:40.562+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:41.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:40.562+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:41.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:40.562+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:41.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:40 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:40.562+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:42.024 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:41.658+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:42.025 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:41.658+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:42.025 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:41.658+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:42.025 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:41.658+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:42.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:41.569+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:42.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:41.569+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:42.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:41.569+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:42.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:41 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:41.570+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:43.024 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:42.646+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:43.025 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:42.646+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:43.025 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:42.646+0000 7f9e642a7700 -1 osd.1 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:34:16.591693+0000 (oldest deadline 2024-03-11T23:34:36.591693+0000) 2024-03-11T23:39:43.025 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1[54472]: 2024-03-11T23:39:42.647+0000 7f9e642a7700 -1 osd.1 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3785 3.b 3:d3ad32af:::100000038d0.00000000:head [write 0~12702 in=12702b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:43.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:42.594+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:43.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:42.594+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:43.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:42.594+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:43.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:42 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:42.595+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:43.964 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:43.615+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:43.965 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:43.615+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:43.965 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:43.615+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:43.965 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:43 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:43.616+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:44.265 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:43 smithi153 podman[85510]: 2024-03-11 23:39:43.966980238 +0000 UTC m=+5.768176278 container died 23e1e01405ed510c2af364a4a0a363f6d8711d399ce16d559aed6c3064b3dc15 (image=quay.ceph.io/ceph-ci/ceph@sha256:ebd35aa0069fa2bbcdd3e20c6fffec19bb7fa44bfa11066e4a72109a40218c32, name=ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=quincy-c186d0c, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T23:39:44.524 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:44 smithi153 podman[85510]: 2024-03-11 23:39:44.383793982 +0000 UTC m=+6.184990013 container cleanup 23e1e01405ed510c2af364a4a0a363f6d8711d399ce16d559aed6c3064b3dc15 (image=quay.ceph.io/ceph-ci/ceph@sha256:ebd35aa0069fa2bbcdd3e20c6fffec19bb7fa44bfa11066e4a72109a40218c32, name=ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=quincy-c186d0c, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-11T23:39:44.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:44 smithi153 bash[85510]: ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1 2024-03-11T23:39:45.024 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:44 smithi153 podman[85528]: 2024-03-11 23:39:44.626598193 +0000 UTC m=+0.656803514 container remove 23e1e01405ed510c2af364a4a0a363f6d8711d399ce16d559aed6c3064b3dc15 (image=quay.ceph.io/ceph-ci/ceph@sha256:ebd35aa0069fa2bbcdd3e20c6fffec19bb7fa44bfa11066e4a72109a40218c32, name=ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-c186d0c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-11T23:39:45.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:44.640+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:45.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:44.641+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:45.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:44.641+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:45.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:44 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:44.641+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:45.328 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:45 smithi153 podman[85618]: 2024-03-11 23:39:44.955543946 +0000 UTC m=+0.019570946 image pull a92afbb1c2834deb6f1bddda3f3c554061dd3ec88e0495c17c1373d9018e466b quay.ceph.io/ceph-ci/ceph@sha256:ebd35aa0069fa2bbcdd3e20c6fffec19bb7fa44bfa11066e4a72109a40218c32 2024-03-11T23:39:45.328 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:45 smithi153 podman[85618]: 2024-03-11 23:39:45.077832387 +0000 UTC m=+0.141859426 container create c925e38dfed6cf75e84ae9c1f4870169660a7619910d75fd684d2254d8546524 (image=quay.ceph.io/ceph-ci/ceph@sha256:ebd35aa0069fa2bbcdd3e20c6fffec19bb7fa44bfa11066e4a72109a40218c32, name=ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-c186d0c, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True) 2024-03-11T23:39:45.329 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:45 smithi153 podman[85618]: 2024-03-11 23:39:45.330820838 +0000 UTC m=+0.394847832 container init c925e38dfed6cf75e84ae9c1f4870169660a7619910d75fd684d2254d8546524 (image=quay.ceph.io/ceph-ci/ceph@sha256:ebd35aa0069fa2bbcdd3e20c6fffec19bb7fa44bfa11066e4a72109a40218c32, name=ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=quincy-c186d0c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-11T23:39:45.603 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:45 smithi153 podman[85618]: 2024-03-11 23:39:45.333884232 +0000 UTC m=+0.397911227 container start c925e38dfed6cf75e84ae9c1f4870169660a7619910d75fd684d2254d8546524 (image=quay.ceph.io/ceph-ci/ceph@sha256:ebd35aa0069fa2bbcdd3e20c6fffec19bb7fa44bfa11066e4a72109a40218c32, name=ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1-deactivate, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=quincy-c186d0c, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T23:39:45.603 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:45 smithi153 podman[85618]: 2024-03-11 23:39:45.439180714 +0000 UTC m=+0.503207716 container attach c925e38dfed6cf75e84ae9c1f4870169660a7619910d75fd684d2254d8546524 (image=quay.ceph.io/ceph-ci/ceph@sha256:ebd35aa0069fa2bbcdd3e20c6fffec19bb7fa44bfa11066e4a72109a40218c32, name=ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1-deactivate, ceph=True, RELEASE=quincy-c186d0c, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-11T23:39:45.965 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:45 smithi153 podman[85618]: 2024-03-11 23:39:45.637340745 +0000 UTC m=+0.701367733 container died c925e38dfed6cf75e84ae9c1f4870169660a7619910d75fd684d2254d8546524 (image=quay.ceph.io/ceph-ci/ceph@sha256:ebd35aa0069fa2bbcdd3e20c6fffec19bb7fa44bfa11066e4a72109a40218c32, name=ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=quincy-c186d0c, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-11T23:39:45.966 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:45.604+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:45.966 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:45.604+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:45.966 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:45.604+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:45.966 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:45 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:45.604+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:46.524 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:46 smithi153 systemd[1]: Stopping Ceph osd.2 for fa913b96-dff9-11ee-95c8-87774f69a715... 2024-03-11T23:39:46.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:46 smithi153 podman[85644]: 2024-03-11 23:39:46.268112034 +0000 UTC m=+0.624388102 container remove c925e38dfed6cf75e84ae9c1f4870169660a7619910d75fd684d2254d8546524 (image=quay.ceph.io/ceph-ci/ceph@sha256:ebd35aa0069fa2bbcdd3e20c6fffec19bb7fa44bfa11066e4a72109a40218c32, name=ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, RELEASE=quincy-c186d0c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-11T23:39:46.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:46 smithi153 systemd[1]: ceph-fa913b96-dff9-11ee-95c8-87774f69a715@osd.1.service: Deactivated successfully. 2024-03-11T23:39:46.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:46 smithi153 systemd[1]: Stopped Ceph osd.1 for fa913b96-dff9-11ee-95c8-87774f69a715. 2024-03-11T23:39:46.525 INFO:journalctl@ceph.osd.1.smithi153.stdout:Mar 11 23:39:46 smithi153 systemd[1]: ceph-fa913b96-dff9-11ee-95c8-87774f69a715@osd.1.service: Consumed 36min 4.856s CPU time. 2024-03-11T23:39:47.024 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:46.599+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:47.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:46.599+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:47.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:46.599+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:47.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:46.600+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:47.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:46.728+0000 7fb4ccee6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T23:39:47.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:46.728+0000 7fb4ccee6700 -1 osd.2 47 *** Got signal Terminated *** 2024-03-11T23:39:47.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:46 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:46.728+0000 7fb4ccee6700 -1 osd.2 47 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T23:39:48.024 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:47.633+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:48.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:47.633+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:48.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:47.633+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:48.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:47 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:47.634+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:49.024 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:48.621+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:49.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:48.621+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:49.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:48.621+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:49.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:48 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:48.622+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:50.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:49.635+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:50.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:49.635+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:50.026 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:49.635+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:50.027 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:49 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:49.636+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:51.024 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:50.605+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:51.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:50.605+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:51.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:50.605+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:51.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:50 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:50.606+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:52.024 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:51.631+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6804 osd.3 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:52.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:51.631+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6812 osd.4 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:52.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:51.631+0000 7fb4cace7700 -1 osd.2 47 heartbeat_check: no reply from 172.21.15.163:6820 osd.5 ever on either front or back, first ping sent 2024-03-11T23:33:58.274733+0000 (oldest deadline 2024-03-11T23:34:18.274733+0000) 2024-03-11T23:39:52.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:51 smithi153 ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2[61797]: 2024-03-11T23:39:51.632+0000 7fb4cace7700 -1 osd.2 47 get_health_metrics reporting 256 slow ops, oldest is osd_op(client.14508.0:3799 3.1e 3:7b489cc7:::20000000004.00000000:head [write 0~1493 in=1493b] snapc 1=[] ondisk+write+known_if_redirected e47) 2024-03-11T23:39:52.702 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:52 smithi153 podman[85722]: 2024-03-11 23:39:52.372690973 +0000 UTC m=+5.757277858 container died 74c3d447a069b723a1a4e37f25517078169f6cfd80b4dbc59e466fccca977e4b (image=quay.ceph.io/ceph-ci/ceph@sha256:ebd35aa0069fa2bbcdd3e20c6fffec19bb7fa44bfa11066e4a72109a40218c32, name=ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2, maintainer=Guillaume Abrioux , RELEASE=quincy-c186d0c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-11T23:39:52.702 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:52 smithi153 podman[85722]: 2024-03-11 23:39:52.704207422 +0000 UTC m=+6.088794308 container cleanup 74c3d447a069b723a1a4e37f25517078169f6cfd80b4dbc59e466fccca977e4b (image=quay.ceph.io/ceph-ci/ceph@sha256:ebd35aa0069fa2bbcdd3e20c6fffec19bb7fa44bfa11066e4a72109a40218c32, name=ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=quincy-c186d0c, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5) 2024-03-11T23:39:53.024 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:52 smithi153 bash[85722]: ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2 2024-03-11T23:39:53.025 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:52 smithi153 podman[85740]: 2024-03-11 23:39:52.913868712 +0000 UTC m=+0.536996577 container remove 74c3d447a069b723a1a4e37f25517078169f6cfd80b4dbc59e466fccca977e4b (image=quay.ceph.io/ceph-ci/ceph@sha256:ebd35aa0069fa2bbcdd3e20c6fffec19bb7fa44bfa11066e4a72109a40218c32, name=ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD, RELEASE=quincy-c186d0c) 2024-03-11T23:39:53.581 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:53 smithi153 podman[85828]: 2024-03-11 23:39:53.213879695 +0000 UTC m=+0.018083142 image pull a92afbb1c2834deb6f1bddda3f3c554061dd3ec88e0495c17c1373d9018e466b quay.ceph.io/ceph-ci/ceph@sha256:ebd35aa0069fa2bbcdd3e20c6fffec19bb7fa44bfa11066e4a72109a40218c32 2024-03-11T23:39:53.582 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:53 smithi153 podman[85828]: 2024-03-11 23:39:53.331366557 +0000 UTC m=+0.135570038 container create 43852d7958f5eef5714b71f8f36ccbddd29f7889dd0b08a129b23a438eb0386f (image=quay.ceph.io/ceph-ci/ceph@sha256:ebd35aa0069fa2bbcdd3e20c6fffec19bb7fa44bfa11066e4a72109a40218c32, name=ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=quincy-c186d0c) 2024-03-11T23:39:53.582 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:53 smithi153 podman[85828]: 2024-03-11 23:39:53.583827765 +0000 UTC m=+0.388031210 container init 43852d7958f5eef5714b71f8f36ccbddd29f7889dd0b08a129b23a438eb0386f (image=quay.ceph.io/ceph-ci/ceph@sha256:ebd35aa0069fa2bbcdd3e20c6fffec19bb7fa44bfa11066e4a72109a40218c32, name=ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=quincy-c186d0c, CEPH_POINT_RELEASE=) 2024-03-11T23:39:53.865 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:53 smithi153 podman[85828]: 2024-03-11 23:39:53.586808589 +0000 UTC m=+0.391012023 container start 43852d7958f5eef5714b71f8f36ccbddd29f7889dd0b08a129b23a438eb0386f (image=quay.ceph.io/ceph-ci/ceph@sha256:ebd35aa0069fa2bbcdd3e20c6fffec19bb7fa44bfa11066e4a72109a40218c32, name=ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=quincy-c186d0c, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True) 2024-03-11T23:39:53.865 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:53 smithi153 podman[85828]: 2024-03-11 23:39:53.658608317 +0000 UTC m=+0.462811777 container attach 43852d7958f5eef5714b71f8f36ccbddd29f7889dd0b08a129b23a438eb0386f (image=quay.ceph.io/ceph-ci/ceph@sha256:ebd35aa0069fa2bbcdd3e20c6fffec19bb7fa44bfa11066e4a72109a40218c32, name=ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, RELEASE=quincy-c186d0c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T23:39:54.160 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:53 smithi153 podman[85828]: 2024-03-11 23:39:53.86737094 +0000 UTC m=+0.671574381 container died 43852d7958f5eef5714b71f8f36ccbddd29f7889dd0b08a129b23a438eb0386f (image=quay.ceph.io/ceph-ci/ceph@sha256:ebd35aa0069fa2bbcdd3e20c6fffec19bb7fa44bfa11066e4a72109a40218c32, name=ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=quincy-c186d0c, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-11T23:39:54.423 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:54 smithi153 podman[85854]: 2024-03-11 23:39:54.403544952 +0000 UTC m=+0.530895811 container remove 43852d7958f5eef5714b71f8f36ccbddd29f7889dd0b08a129b23a438eb0386f (image=quay.ceph.io/ceph-ci/ceph@sha256:ebd35aa0069fa2bbcdd3e20c6fffec19bb7fa44bfa11066e4a72109a40218c32, name=ceph-fa913b96-dff9-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=quincy-c186d0c, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T23:39:54.774 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:54 smithi153 systemd[1]: ceph-fa913b96-dff9-11ee-95c8-87774f69a715@osd.2.service: Deactivated successfully. 2024-03-11T23:39:54.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:54 smithi153 systemd[1]: Stopped Ceph osd.2 for fa913b96-dff9-11ee-95c8-87774f69a715. 2024-03-11T23:39:54.775 INFO:journalctl@ceph.osd.2.smithi153.stdout:Mar 11 23:39:54 smithi153 systemd[1]: ceph-fa913b96-dff9-11ee-95c8-87774f69a715@osd.2.service: Consumed 18min 16.583s CPU time. 2024-03-11T23:39:57.422 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:39:57.423 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:40:00.478 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:40:09.479 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:40:09.480 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:40:12.546 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:40:24.547 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:40:24.548 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:40:27.614 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:40:30.615 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:40:30.617 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:40:30.690 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:40:30.690 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-11T23:40:30.691 INFO:tasks.cephadm:Teardown complete 2024-03-11T23:40:30.691 ERROR:teuthology.run_tasks:Manager failed: cephadm Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/cephadm.py", line 1553, in distribute_config_and_admin_keyring yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/cephadm.py", line 1807, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/cephadm.py", line 1810, in task log.info('Teardown begin') File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/cephadm.py", line 1686, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/cephadm.py", line 142, in download_cephadm _rm_cluster(ctx, cluster_name) File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/cephadm.py", line 310, in _rm_cluster ctx.cluster.run(args=[ File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi163 2024-03-11T23:40:30.713 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-11T23:40:30.724 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-03-11T23:40:30.741 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-11T23:40:30.741 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-11T23:40:30.756 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:40:30.757 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:40:33.822 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:40:42.824 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:40:42.825 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:40:45.890 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:40:57.891 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:40:57.892 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:41:00.958 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:41:03.959 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:41:03.960 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:41:04.030 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:41:04.030 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-11T23:41:04.081 ERROR:teuthology.run_tasks:Manager failed: install Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/util.py", line 148, in ship_utilities yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 220, in install yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/util.py", line 150, in ship_utilities _remove_utilities(ctx, filenames) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/util.py", line 121, in _remove_utilities ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi163 2024-03-11T23:41:04.101 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-11T23:41:04.118 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-11T23:41:04.118 DEBUG:teuthology.orchestra.run.smithi153:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T23:41:04.126 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:41:04.127 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:41:04.141 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-03-11T23:41:04.145 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T23:41:04.145 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-03-11T23:41:04.145 INFO:teuthology.orchestra.run.smithi153.stdout:^- hv01.front.sepia.ceph.com 2 6 377 1 +2868us[+2868us] +/- 96ms 2024-03-11T23:41:04.145 INFO:teuthology.orchestra.run.smithi153.stdout:^* hv02.front.sepia.ceph.com 3 6 377 8 -249us[ -169us] +/- 65ms 2024-03-11T23:41:04.145 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 2 +554us[ +554us] +/- 74ms 2024-03-11T23:41:04.146 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T23:41:07.198 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:41:16.199 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:41:16.200 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:41:19.262 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:41:31.264 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:41:31.265 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:41:34.334 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:41:37.335 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:41:37.336 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:41:37.406 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:41:37.406 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-11T23:41:37.407 ERROR:teuthology.run_tasks:Manager failed: clock Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/clock.py", line 66, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/clock.py", line 72, in task cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi163 2024-03-11T23:41:37.424 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-11T23:41:37.436 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-11T23:41:37.437 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-11T23:41:37.486 DEBUG:teuthology.orchestra.run.smithi153:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T23:41:37.573 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:41:37.574 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:41:40.638 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:41:49.640 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:41:49.640 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:41:52.702 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:42:04.703 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:42:04.704 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:42:07.774 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:42:10.776 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:42:10.776 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:42:10.846 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:42:10.847 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-11T23:42:10.847 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi163 2024-03-11T23:42:10.856 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-11T23:42:10.868 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-11T23:42:10.881 INFO:teuthology.task.internal:Duration was 3436.385015 seconds 2024-03-11T23:42:10.882 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-11T23:42:10.896 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-11T23:42:10.896 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T23:42:10.902 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:42:10.902 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:42:10.949 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T23:42:13.982 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:42:22.984 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:42:22.987 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:42:26.046 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:42:38.047 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:42:38.048 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:42:41.118 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:42:44.119 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:42:44.120 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:42:44.194 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:42:44.194 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-11T23:42:44.195 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/syslog.py", line 85, in syslog cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi163 2024-03-11T23:42:44.213 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-11T23:42:44.226 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-11T23:42:44.226 DEBUG:teuthology.orchestra.run.smithi153:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T23:42:44.263 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:42:44.263 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:42:47.326 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:42:56.330 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:42:56.382 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:42:59.454 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:43:11.455 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:43:11.456 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:43:14.526 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:43:17.527 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:43:17.528 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:43:17.598 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:43:17.598 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-11T23:43:17.599 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi163 2024-03-11T23:43:17.615 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-11T23:43:17.629 DEBUG:teuthology.orchestra.run.smithi153:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T23:43:17.636 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:43:17.636 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:43:17.664 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = core 2024-03-11T23:43:20.702 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:43:29.703 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:43:29.704 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:43:32.766 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:43:44.768 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:43:44.769 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:43:47.838 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:43:50.839 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:43:50.840 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:43:50.914 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:43:50.914 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-11T23:43:50.915 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 492, in coredump cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi163 2024-03-11T23:43:50.930 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-11T23:43:50.944 INFO:teuthology.task.internal:Transferring archived files... 2024-03-11T23:43:50.944 DEBUG:teuthology.misc:Transferring archived files from smithi153:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592012/remote/smithi153 2024-03-11T23:43:50.945 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T23:43:50.987 DEBUG:teuthology.misc:Transferring archived files from smithi163:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592012/remote/smithi163 2024-03-11T23:43:50.987 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:43:50.987 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:43:54.046 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:44:03.047 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:44:03.048 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:44:06.110 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:44:18.111 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:44:18.112 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:44:21.182 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:44:24.184 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:44:24.184 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:44:24.254 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:44:24.254 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-11T23:44:24.255 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 748, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True, compress=False) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 678, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi163 2024-03-11T23:44:24.273 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-11T23:44:24.290 INFO:teuthology.task.internal:Not uploading archives. 2024-03-11T23:44:24.290 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-11T23:44:24.302 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-11T23:44:24.302 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T23:44:24.309 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:44:24.310 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:44:24.325 INFO:teuthology.orchestra.run.smithi153.stdout: 262166 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 11 23:40 /home/ubuntu/cephtest 2024-03-11T23:44:24.325 INFO:teuthology.orchestra.run.smithi153.stdout: 262167 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 11 23:43 /home/ubuntu/cephtest/archive 2024-03-11T23:44:24.325 INFO:teuthology.orchestra.run.smithi153.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 22:44 /home/ubuntu/cephtest/archive/syslog 2024-03-11T23:44:24.325 INFO:teuthology.orchestra.run.smithi153.stdout: 265301 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Mar 11 22:44 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T23:44:24.325 INFO:teuthology.orchestra.run.smithi153.stdout: 265300 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Mar 11 22:44 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T23:44:24.325 INFO:teuthology.orchestra.run.smithi153.stdout: 402361 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 23:41 /home/ubuntu/cephtest/archive/audit 2024-03-11T23:44:24.326 INFO:teuthology.orchestra.run.smithi153.stdout: 426300 176 -rw------- 1 ubuntu ubuntu 179920 Mar 11 23:41 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-03-11T23:44:24.326 INFO:teuthology.orchestra.run.smithi153.stdout: 265386 364 -rwxr-xr-x 1 ubuntu ubuntu 372315 Mar 11 22:52 /home/ubuntu/cephtest/cephadm 2024-03-11T23:44:24.327 INFO:teuthology.orchestra.run.smithi153.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-03-11T23:44:27.390 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:44:36.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:44:36.392 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:44:39.458 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:44:51.459 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:44:51.460 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:44:54.526 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:44:57.527 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-11T23:44:57.528 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T23:44:57.598 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-11T23:44:57.598 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-11T23:44:57.599 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 54, in base ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi163 2024-03-11T23:44:57.615 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-11T23:44:57.629 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-11T23:44:57.645 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: OSError: [Errno 113] No route to host 2024-03-11T23:44:57.646 INFO:teuthology.nuke:Checking targets against current locks 2024-03-11T23:44:57.671 DEBUG:teuthology.nuke:shortname: smithi153 2024-03-11T23:44:57.671 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T23:44:57.692 DEBUG:teuthology.nuke:shortname: smithi163 2024-03-11T23:44:57.692 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T23:44:57.696 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592012', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 22:32:44.715334', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGSU9hxjSkRd+UPEzk4H3VymcwJcjPDWVRnPgi+lAJGEQjMpSPys/9/UtPTw/AXe3s9bte0+1Ft3vbdE2ZqzApo='} 2024-03-11T23:44:57.721 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi163.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592012', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 22:32:44.713856', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBr+LqZ81CgxZsYI9J50PTe8gyFZUpqQwnzD6uUcbAGvGTOHbEWwPUxcjZEYY+qazT5YfadfBUdAIuqqrVBb3Bw='} 2024-03-11T23:44:57.750 INFO:teuthology.orchestra.console.smithi153:Power off 2024-03-11T23:44:57.750 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T23:44:57.783 DEBUG:teuthology.orchestra.console.smithi153:power off output: Chassis Power Control: Down/Off 2024-03-11T23:44:57.783 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T23:44:57.804 INFO:teuthology.orchestra.console.smithi163:Power off 2024-03-11T23:44:57.804 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T23:44:57.817 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-03-11T23:44:57.829 DEBUG:teuthology.orchestra.console.smithi163:power off output: Chassis Power Control: Down/Off 2024-03-11T23:44:57.829 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T23:44:57.855 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is on 2024-03-11T23:45:01.819 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T23:45:01.857 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T23:45:01.932 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-03-11T23:45:01.968 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is on 2024-03-11T23:45:05.934 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T23:45:05.969 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T23:45:06.045 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is off 2024-03-11T23:45:06.081 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is off 2024-03-11T23:45:06.147 INFO:teuthology.orchestra.console.smithi153:Power off completed 2024-03-11T23:45:06.182 INFO:teuthology.orchestra.console.smithi163:Power off completed 2024-03-11T23:45:06.288 INFO:teuthology.run:Summary data: description: orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 3436.3850152492523 failure_reason: 'Command failed on smithi153 with status 1: ''sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fa913b96-dff9-11ee-95c8-87774f69a715 -e sha1=94a539707733325322964c6a187c6a1b41be1daf -- bash -c \''ceph versions | jq -e \''"\''"\''.mgr | keys\''"\''"\'' | grep $sha1\''''' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=91c4299b4a3b4e8fbda1931f0337924d status: fail success: false 2024-03-11T23:45:06.288 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-11T23:45:06.359 INFO:teuthology.run:FAIL